Skip to content

Commit d351c67

Browse files
kevin-dpcursoragentautofix-ci[bot]claude
authored
feat: SQLite persistence core (#1358)
* docs: add sqlite persistence plan and phased guard rails * feat: add sqlite persisted collection core phase-0 package * feat(db): add index lifecycle events and removeIndex API * ci: apply automated fixes * feat(db): harden index lifecycle contract for phase-2 bootstrap * test(db): align index metadata expectations with canonicalization * feat: implement phase 2 persisted wrapper runtime * ci: apply automated fixes * fix: stabilize phase 2 hydration and stream typing * ci: apply automated fixes * fix: harden phase 2 consistency and recovery paths * ci: apply automated fixes * feat: add sqlite core persistence adapter for phase 3 * fix: address phase 3 adapter lint and test regressions * ci: apply automated fixes * feat: harden phase 3 sqlite adapter semantics and contracts * fix: resolve relational comparator typing in sqlite adapter * test: extend phase 3 sqlite adapter contract coverage * ci: apply automated fixes * feat: add time-based applied_tx pruning policy * fix: align sqlite fallback equality with query semantics * fix: support alias-qualified paths in fallback evaluator * refactor: reuse query evaluator in sqlite adapter fallback * ci: apply automated fixes * feat: add node sqlite persisted collection package * fix: add better-sqlite3 type definitions for node adapter * test: extract reusable sqlite runtime contract suites * fix: tighten runtime contract typing in node tests * test: extract phase-3 sqlite adapter contract suite * test: exclude contract module from vitest test discovery * feat: add electron sqlite ipc bridge package * ci: apply automated fixes * fix: tighten electron ipc protocol and contract typing * fix: simplify electron ipc envelope typing * ci: apply automated fixes * fix: stabilize electron renderer request typing * test: add portable runtime bridge e2e contract * ci: apply automated fixes * fix: stabilize electron runtime bridge e2e harness * ci: apply automated fixes * test: run electron runtime e2e via built fixtures * ci: apply automated fixes * test: split electron e2e vitest config * fix: make electron e2e vitest config standalone * test: build core db package before electron e2e * test: build db-ivm before electron runtime e2e * test: include stderr output on electron e2e timeout * fix: avoid node native dependency in electron e2e harness * ci: apply automated fixes * fix: lazy-load node adapter without import-meta * fix: load file-based renderer page in electron e2e * fix: run electron e2e under xvfb without headless * fix: disable dev-shm usage for electron e2e * fix: use cjs preload for electron runtime bridge e2e * test: capture renderer diagnostics in electron e2e runner * fix: resolve preload renderer module via absolute path * fix: disable sandbox for electron e2e preload * test: exclude e2e fixtures from package typecheck scope * test: add optional full-suite electron e2e mode * ci: apply automated fixes * fix: support multi-collection contract runs in e2e mode * fix: allow graceful electron shutdown after e2e result * fix: avoid WAL sidecar drift in full electron e2e mode * ci: apply automated fixes * fix: support bigint and typed date comparisons in sqlite core * test: add portable persisted collection conformance harness * ci: apply automated fixes * fix: harden typed conformance harness and rollback test * ci: apply automated fixes * fix: stabilize persisted cursor paging and node conformance harness * test: add shared conformance suite for electron persisted * ci: apply automated fixes * fix: harden electron full-e2e transport and timing * ci: apply automated fixes * fix: preserve bigint payloads across electron fixture bridge * ci: apply automated fixes * fix: serialize electron e2e results and extend full-mode timeouts * fix: raise electron full-mode test time budgets via vitest config * ci: apply automated fixes * fix: use function-safe e2e input encoding for electron bridge * fix: strip subscription handles from electron loadSubset rpc * fix: preserve shared ir nodes in electron e2e input encoder * fix: make electron main require base path cjs-compatible * test: serialize full-mode electron suites and align ipc timeouts * ci: apply automated fixes * test: isolate electron fixture env from vitest coverage vars * test: increase full-mode electron rpc time budgets * fix: resolve electron require base path from absolute argv entry * feat: add react-native/expo sqlite persisted collection package * ci: apply automated fixes * fix: align mobile adapter typing and burst persistence test * ci: apply automated fixes * fix(react-native): harden transactions and expand parity coverage * ci: apply automated fixes * test(react-native): stabilize lifecycle and runtime contract coverage * ci: apply automated fixes * fix(react-native): resolve transaction deadlock and harden runtime test rails * ci: apply automated fixes * fix(sqlite-driver): enforce transaction callback parity across runtimes * ci: apply automated fixes * test(react-native): add expo lifecycle coverage and ci e2e wiring * docs(ci): fix mobile setup examples and enforce runtime lane * ci: apply automated fixes * ci(e2e): make mobile runtime lane enforceable but non-blocking by default * ci(e2e): run node and electron persisted suites * feat: add cloudflare DO sqlite persistence with wrangler e2e * test: harden cloudflare DO schema and transaction semantics * docs: add complete API readmes for sqlite persistence packages * refactor: unify sqlite persistence APIs around shared persisters * fix: preserve adapter defaults while keeping mode-aware persistence * refactor: simplify sqlite persistence APIs across runtimes * fix: satisfy generic cast in node persistence * fix: narrow generic adapter cast in mobile persistence * fix: narrow generic adapter cast in cloudflare persistence * fix: tighten electron test harness and renderer typing * fix: tighten electron ipc test persistence typing * fix: normalize markIndexRemoved return type in electron tests * fix(cloudflare-e2e): use collection-resolved adapter in worker fixture * fix(electron): preserve remote error code in renderer IPC * fix(electron): catch async adapter errors in main IPC bridge * feat(sqlite): accept native runtime handles in persistence factories * refactor(persistence): require native sqlite handles only * chore: refresh pnpm lockfile for cloudflare package peers * ci: apply automated fixes * feat(browser): implement phase 7 single-tab persistence * ci: apply automated fixes * fix(browser): move phase 7 opfs path into worker * ci: apply automated fixes * fix(persistence): restore stream position on startup to prevent duplicate tx skipping The PersistedCollectionRuntime never restored its stream position from the database on startup, always beginning at localTerm=1, localSeq=0. After a page reload, the first new mutation would collide with a previously applied transaction (term=1, seq=1), causing the SQLite adapter's applyCommittedTx to silently skip it as a duplicate. Add getStreamPosition to PersistenceAdapter (optional) and implement it in SQLiteCorePersistenceAdapter. Call it from startInternal() so that observeStreamPosition seeds the local counters before any mutations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(persistence): inline row data in tx:committed broadcasts Include full row values in the tx:committed message so receiving tabs can apply changes directly without a SQLite round-trip via loadRowsByKeys. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: apply automated fixes * ci: apply automated fixes (attempt 2/3) * docs: update phase 8 plan with implementation status annotations Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: apply automated fixes * feat(browser): implement Phase 8 BrowserCollectionCoordinator for multi-tab support Web Locks for per-collection leadership election, BroadcastChannel for cross-tab RPC transport, DB writer lock for SQLite write serialization, envelope dedup for exactly-once mutations, and leader heartbeats. Includes 15 unit tests with Web Locks/BroadcastChannel mocks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: apply automated fixes * fix(browser): prevent stuck leadership state on acquireLeadership errors Previously, `state.isLeader = true` was set before the setup code that calls `getStreamPosition()`. If `getStreamPosition` threw (e.g. due to a UNIQUE constraint violation from React StrictMode double-mounting), `isLeader` remained permanently stuck at `true` because the `finally` block that resets it was inside an inner try/finally that was never entered. Fix: Wrap the entire lock callback body in a single try/finally. Set `state.isLeader = true` only after successful setup (stream position restore and term increment). The finally block always runs and resets `isLeader = false` + cleans up the heartbeat timer. Also refactors the coordinator to support lazy adapter wiring via `setAdapter()`, allowing `createBrowserWASQLitePersistence` to inject the adapter after construction. This enables the demo to construct the coordinator without requiring the adapter upfront. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(persistence): always route mutations through coordinator to prevent seq collisions The leader tab had two mutation paths: a "direct" path (write to SQLite and broadcast) and an RPC path (through the coordinator). Previously, only follower tabs used the RPC path — the leader bypassed the coordinator and wrote directly. This caused a seq collision: the leader's direct writes incremented the runtime's `localSeq` but left the coordinator's `state.latestSeq` at 0. When a follower later sent an RPC, the coordinator assigned seq starting from 1 again, producing duplicate seq numbers. The leader then skipped these "already-seen" tx:committed messages, causing follower mutations to silently disappear. Fix: Always route through `requestApplyLocalMutations` when available, regardless of leader/follower status. This keeps the coordinator's seq counter in sync with all writes. Also removes `requestApplyLocalMutations` from `SingleProcessCoordinator` — it was a stub that returned success without persisting, which would break now that the leader uses this path. Single-process mode correctly falls back to the direct path since it has no multi-tab coordination. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(persistence): allow leader to process coordinator-delivered tx:committed messages The leader tab's `onCoordinatorMessage` handler skipped ALL messages where `senderId` matched the coordinator's own node ID. But when the coordinator processes a follower's RPC in `handleApplyLocalMutations`, it delivers the resulting `tx:committed` to local subscribers using the coordinator's own `senderId`. This caused the leader's runtime to silently ignore follower mutations — they were written to SQLite but never applied to the leader's in-memory collection. Fix: Allow `tx:committed` messages from self to pass through the filter. The seq dedup logic in `processCommittedTxUnsafe` already prevents double-processing: when the leader's own mutations go through the coordinator, `observeStreamPosition` is called with the response's term/seq before the local `tx:committed` delivery runs under the mutex, so the duplicate is detected via `txCommitted.seq <= this.latestSeq`. Other message types (heartbeats, resets) from self are still skipped. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: apply automated fixes * fix(persistence): handle delete messages with value in normalizeSyncWriteMessage When queryCollectionOptions detects a server-side deletion, it sends { type: 'delete', value: oldItem } through the sync. The persistence layer only checked for 'key' in message to detect deletes, causing value-based deletes to be misclassified as updates. Also use optional chaining for process.versions in React Native where process exists but versions may not. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(electron-persistence): add ElectronCollectionCoordinator for cross-window sync Add ElectronCollectionCoordinator using BroadcastChannel + Web Locks for leader election and cross-window coordination in Electron renderer windows. Wire coordinator into renderer persistence via setAdapter(), add getStreamPosition to the IPC protocol, and export from package index. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: apply automated fixes * chore: extract electron, node, and cloudflare persistence packages to follow-up branches Remove db-electron-sqlite-persisted-collection, db-node-sqlite-persisted-collection, and db-cloudflare-do-sqlite-persisted-collection from this branch so it contains only the core persistence packages (db, db-sqlite-persisted-collection-core, db-browser-wa-sqlite-persisted-collection, db-react-native-sqlite-persisted-collection). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): remove extra blank line in e2e-tests workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Delete plan * Changeset * Strip virtual props from persistence tests * Changeset fix * Formatting --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 43ecbfa commit d351c67

73 files changed

Lines changed: 14128 additions & 25 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.changeset/sqlite-persistence.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
'@tanstack/db': patch
3+
'@tanstack/db-sqlite-persisted-collection-core': patch
4+
'@tanstack/db-browser-wa-sqlite-persisted-collection': patch
5+
'@tanstack/db-react-native-sqlite-persisted-collection': patch
6+
---
7+
8+
feat(persistence): add SQLite-based offline persistence for collections
9+
10+
Adds a new persistence layer that durably stores collection data in SQLite, enabling applications to survive page reloads and app restarts.
11+
12+
**Core persistence (`@tanstack/db-sqlite-persisted-collection-core`)**
13+
14+
- New package providing the shared SQLite persistence runtime: hydration, streaming, transaction tracking, and applied-tx pruning
15+
- SQLite core adapter with full query compilation, index management, and schema migration support
16+
- Portable conformance test contracts for runtime-specific adapters
17+
18+
**Browser (`@tanstack/db-browser-wa-sqlite-persisted-collection`)**
19+
20+
- New package for browser persistence via wa-sqlite backed by OPFS
21+
- Single-tab persistence with OPFS-based SQLite storage
22+
- `BrowserCollectionCoordinator` for multi-tab leader-election and cross-tab sync
23+
24+
**React Native (`@tanstack/db-react-native-sqlite-persisted-collection`)**
25+
26+
- New package for React Native persistence via op-sqlite
27+
- Adapter with transaction deadlock prevention and runtime parity coverage

.github/workflows/e2e-tests.yml

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ jobs:
99
e2e-tests:
1010
name: Run E2E Tests
1111
runs-on: ubuntu-latest
12-
timeout-minutes: 10
12+
timeout-minutes: 35
1313

1414
steps:
1515
- name: Checkout code
@@ -62,6 +62,43 @@ jobs:
6262
env:
6363
ELECTRIC_URL: http://localhost:3000
6464

65+
- name: Run React Native/Expo persisted collection E2E tests
66+
run: |
67+
cd packages/db-react-native-sqlite-persisted-collection
68+
pnpm test:e2e
69+
70+
- name: Run React Native/Expo runtime E2E lane
71+
run: |
72+
is_fork_pr=false
73+
require_runtime_lane=${TANSTACK_DB_REQUIRE_MOBILE_RUNTIME_LANE:-0}
74+
if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then
75+
is_fork_pr=$(jq -r '.pull_request.head.repo.fork' "${GITHUB_EVENT_PATH}")
76+
fi
77+
78+
if [ "${is_fork_pr}" = "true" ]; then
79+
echo "Skipping runtime mobile lane for fork PR (repo vars are unavailable)."
80+
exit 0
81+
fi
82+
83+
if [ -z "${TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE}" ]; then
84+
if [ "${require_runtime_lane}" = "1" ]; then
85+
echo "::error::Missing repository variable TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE while TANSTACK_DB_REQUIRE_MOBILE_RUNTIME_LANE=1."
86+
echo "::error::Set TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE (and optional TANSTACK_DB_MOBILE_SQLITE_FACTORY_EXPORT)."
87+
exit 1
88+
fi
89+
90+
echo "Skipping runtime mobile lane (no TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE configured)."
91+
echo "Set TANSTACK_DB_REQUIRE_MOBILE_RUNTIME_LANE=1 to enforce this lane."
92+
exit 0
93+
fi
94+
95+
cd packages/db-react-native-sqlite-persisted-collection
96+
pnpm test:e2e:runtime
97+
env:
98+
TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE: ${{ vars.TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE }}
99+
TANSTACK_DB_MOBILE_SQLITE_FACTORY_EXPORT: ${{ vars.TANSTACK_DB_MOBILE_SQLITE_FACTORY_EXPORT }}
100+
TANSTACK_DB_REQUIRE_MOBILE_RUNTIME_LANE: ${{ vars.TANSTACK_DB_REQUIRE_MOBILE_RUNTIME_LANE }}
101+
65102
- name: Stop Docker services
66103
if: always()
67104
run: |

package.json

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@
1010
"pnpm": {
1111
"overrides": {
1212
"metro": "0.82.5"
13-
}
13+
},
14+
"onlyBuiltDependencies": [
15+
"better-sqlite3",
16+
"electron"
17+
]
1418
},
1519
"scripts": {
1620
"build": "pnpm --filter \"./packages/**\" build",
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# @tanstack/db-browser-wa-sqlite-persisted-collection
2+
3+
Thin browser SQLite persistence for TanStack DB using `wa-sqlite` + OPFS.
4+
5+
## Public API
6+
7+
- `createBrowserWASQLitePersistence(...)`
8+
- `openBrowserWASQLiteOPFSDatabase(...)`
9+
- `persistedCollectionOptions(...)` (re-exported from core)
10+
11+
## Quick start
12+
13+
```ts
14+
import { createCollection } from '@tanstack/db'
15+
import {
16+
createBrowserWASQLitePersistence,
17+
openBrowserWASQLiteOPFSDatabase,
18+
persistedCollectionOptions,
19+
} from '@tanstack/db-browser-wa-sqlite-persisted-collection'
20+
21+
type Todo = {
22+
id: string
23+
title: string
24+
completed: boolean
25+
}
26+
27+
const database = await openBrowserWASQLiteOPFSDatabase({
28+
databaseName: `tanstack-db.sqlite`,
29+
})
30+
31+
const persistence = createBrowserWASQLitePersistence({
32+
database,
33+
})
34+
35+
export const todosCollection = createCollection(
36+
persistedCollectionOptions<Todo, string>({
37+
id: `todos`,
38+
getKey: (todo) => todo.id,
39+
persistence,
40+
schemaVersion: 1, // Per-collection schema version
41+
}),
42+
)
43+
```
44+
45+
## Notes
46+
47+
- This package is Phase 7 single-tab browser wiring: it uses
48+
`SingleProcessCoordinator` semantics by default.
49+
- `openBrowserWASQLiteOPFSDatabase(...)` starts a dedicated Web Worker and
50+
routes SQL operations through it. OPFS sync access handle APIs are used in
51+
that worker context.
52+
- Single-tab mode does not require BroadcastChannel or Web Locks for
53+
correctness.
54+
- OPFS capability failures are surfaced as `PersistenceUnavailableError`.

0 commit comments

Comments
 (0)