Skip to content

eth-go-producers

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

eth-go-producers

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
cmd/ingest
cmd/ingest
 
 
pkg
pkg
 
 
.env.example
.env.example
 
 
.gitignore
.gitignore
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
docker-compose.yml
docker-compose.yml
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 

README.md

Go RPC to Kafka Ingestion Service

Ethereum data ingestion service written in Go. It fetches blocks, transactions, logs, and token transfers from an Ethereum RPC node and produces them to Kafka topics using Protobuf serialization.

Features

  • Resilient RPC Client: Retries with exponential backoff, rate limiting.
  • High Throughput: Concurrent worker pool for block processing.
  • Data Integrity: Idempotent Kafka production, Schema Registry integration.
  • State Management: Ordered checkpointing, reorg detection and rewinding.
  • Observability: Prometheus metrics.

Architecture

  • cmd/ingest: Application entrypoint.
  • pkg/common: Shared utilities (Config, Logger, Metrics).
  • pkg/adapter: External adapters (RPC, Kafka).
  • pkg/engine: Core logic (Ingest Manager, Worker Pool, Checkpoint Store, Mapper, Token Cache).
  • pkg/proto: Generated Protobuf code.

Prerequisites

  • Go 1.25+
  • Kafka Cluster
  • Confluent Schema Registry
  • Ethereum Node (RPC/WSS)

Configuration

Copy .env.example to .env and configure:

cp .env.example .env

Build & Run

make build
make run

Metrics

Metrics are exposed at http://localhost:9090/metrics.