[Prototype] Bigtable

[Kbhat1]

Describe your changes and provide context

Testing performed to validate your change

---

Note

Medium Risk
Adds an optional fallback path for pruned Get/Has state-store reads to query external Scylla/Cassandra or Bigtable, plus new ingestion tooling and cloud/db dependencies; misconfiguration or backend issues could affect historical query behavior.

Overview
Adds prototype historical-state offload support so SS point reads (Get/Has) for versions below local retention can fall back to an external store (Scylla/Cassandra or Bigtable), wrapped via historical.FallbackStateStore and enabled through new [state-store] TOML/flag config (mutually exclusive backends).

Introduces a new Kafka-based offload consumer (sei-db/state_db/ss/offload/consumer) with configurable sinks for Scylla and Bigtable, including schema/examples and retry/batching/backpressure behavior, and exports offload.NewSASLMechanism so external consumers share the producer auth path.

Extends SS config structs/templates/tests for the new historical-offload settings and adds new dependencies (cloud.google.com/go/bigtable, github.com/gocql/gocql, golang.org/x/oauth2).

^{Reviewed by Cursor Bugbot for commit 66f8c9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	May 28, 2026, 2:45 PM

[codecov]

Codecov Report

❌ Patch coverage is 41.82243% with 747 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.15%. Comparing base (654d40b) to head (b3e0b81).
⚠️ Report is 69 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/offload/historical/bigtable.go	38.52%	214 Missing and 19 partials ⚠️
sei-db/state_db/ss/offload/consumer/consumer.go	0.00%	177 Missing ⚠️
sei-db/state_db/ss/offload/historical/scylla.go	50.68%	71 Missing and 1 partial ⚠️
sei-db/state_db/ss/offload/consumer/scylla.go	62.17%	48 Missing and 11 partials ⚠️
sei-db/state_db/ss/offload/historical/store.go	53.76%	34 Missing and 9 partials ⚠️
sei-db/state_db/ss/store.go	22.64%	39 Missing and 2 partials ⚠️
sei-db/state_db/ss/offload/consumer/config.go	39.28%	29 Missing and 5 partials ⚠️
sei-db/state_db/ss/offload/consumer/bigtable.go	69.72%	27 Missing and 6 partials ⚠️
sei-db/state_db/ss/offload/consumer/kafka.go	44.82%	31 Missing and 1 partial ⚠️
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00%	21 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3437      +/-   ##
==========================================
- Coverage   59.25%   59.15%   -0.11%     
==========================================
  Files        2110     2121      +11     
  Lines      174181   176527    +2346     
==========================================
+ Hits       103210   104417    +1207     
- Misses      62044    63102    +1058     
- Partials     8927     9008      +81     

Flag	Coverage Δ
sei-chain-pr	51.45% <100.00%> (?)
sei-db	70.41% <ø> (ø)
sei-db-state-db-pr	41.70% <41.59%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/seidb.go	79.04% <100.00%> (+7.04%)	⬆️
sei-db/config/ss_config.go	100.00% <ø> (ø)
sei-db/state_db/ss/offload/kafka.go	59.37% <100.00%> (ø)
sei-db/state_db/ss/offload/consumer/compact.go	93.10% <93.10%> (ø)
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00% <0.00%> (ø)
sei-db/state_db/ss/offload/consumer/kafka.go	44.82% <44.82%> (ø)
sei-db/state_db/ss/offload/consumer/bigtable.go	69.72% <69.72%> (ø)
sei-db/state_db/ss/offload/consumer/config.go	39.28% <39.28%> (ø)
sei-db/state_db/ss/store.go	24.07% <22.64%> (-75.93%)	⬇️
sei-db/state_db/ss/offload/historical/store.go	53.76% <53.76%> (ø)
... and 4 more

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

High Risk
Changes core state-store read behavior for pruned heights and introduces external DB/Kafka dependencies; misconfiguration or stale offload data could return wrong historical state.

Overview
Adds a prototype historical state offload path so pruned local state store (SS) versions can still be served for point Get/Has from ScyllaDB/Cassandra or Google Bigtable, while local SS remains authoritative for recent data, writes, pruning, and iterators.

Node integration: NewStateStore optionally wraps the composite SS in FallbackStateStore, which delegates to an external reader when the requested version is below GetEarliestVersion(). Only one fallback backend may be configured. New [state-store] / StateStoreConfig fields and TOML document Scylla and Bigtable settings; app flags and tests cover Scylla wiring.

Ingest: A Kafka consumer (historical-scylla-consumer) batches changelog entries, compacts per-key mutations within a version, writes mutation rows then version markers (Scylla CQL schema + Bigtable salted row keys), and commits offsets only after successful sink writes. Shared historical package implements readers/writers; NewSASLMechanism is exported for consumer Kafka auth.

Dependencies: gocql, cloud.google.com/go/bigtable, and related indirect modules.

^{Reviewed by Cursor Bugbot for commit b3e0b81. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ecdf5cc. Configure here.}