Commit d351c67
feat: SQLite persistence core (#1358)
* docs: add sqlite persistence plan and phased guard rails
* feat: add sqlite persisted collection core phase-0 package
* feat(db): add index lifecycle events and removeIndex API
* ci: apply automated fixes
* feat(db): harden index lifecycle contract for phase-2 bootstrap
* test(db): align index metadata expectations with canonicalization
* feat: implement phase 2 persisted wrapper runtime
* ci: apply automated fixes
* fix: stabilize phase 2 hydration and stream typing
* ci: apply automated fixes
* fix: harden phase 2 consistency and recovery paths
* ci: apply automated fixes
* feat: add sqlite core persistence adapter for phase 3
* fix: address phase 3 adapter lint and test regressions
* ci: apply automated fixes
* feat: harden phase 3 sqlite adapter semantics and contracts
* fix: resolve relational comparator typing in sqlite adapter
* test: extend phase 3 sqlite adapter contract coverage
* ci: apply automated fixes
* feat: add time-based applied_tx pruning policy
* fix: align sqlite fallback equality with query semantics
* fix: support alias-qualified paths in fallback evaluator
* refactor: reuse query evaluator in sqlite adapter fallback
* ci: apply automated fixes
* feat: add node sqlite persisted collection package
* fix: add better-sqlite3 type definitions for node adapter
* test: extract reusable sqlite runtime contract suites
* fix: tighten runtime contract typing in node tests
* test: extract phase-3 sqlite adapter contract suite
* test: exclude contract module from vitest test discovery
* feat: add electron sqlite ipc bridge package
* ci: apply automated fixes
* fix: tighten electron ipc protocol and contract typing
* fix: simplify electron ipc envelope typing
* ci: apply automated fixes
* fix: stabilize electron renderer request typing
* test: add portable runtime bridge e2e contract
* ci: apply automated fixes
* fix: stabilize electron runtime bridge e2e harness
* ci: apply automated fixes
* test: run electron runtime e2e via built fixtures
* ci: apply automated fixes
* test: split electron e2e vitest config
* fix: make electron e2e vitest config standalone
* test: build core db package before electron e2e
* test: build db-ivm before electron runtime e2e
* test: include stderr output on electron e2e timeout
* fix: avoid node native dependency in electron e2e harness
* ci: apply automated fixes
* fix: lazy-load node adapter without import-meta
* fix: load file-based renderer page in electron e2e
* fix: run electron e2e under xvfb without headless
* fix: disable dev-shm usage for electron e2e
* fix: use cjs preload for electron runtime bridge e2e
* test: capture renderer diagnostics in electron e2e runner
* fix: resolve preload renderer module via absolute path
* fix: disable sandbox for electron e2e preload
* test: exclude e2e fixtures from package typecheck scope
* test: add optional full-suite electron e2e mode
* ci: apply automated fixes
* fix: support multi-collection contract runs in e2e mode
* fix: allow graceful electron shutdown after e2e result
* fix: avoid WAL sidecar drift in full electron e2e mode
* ci: apply automated fixes
* fix: support bigint and typed date comparisons in sqlite core
* test: add portable persisted collection conformance harness
* ci: apply automated fixes
* fix: harden typed conformance harness and rollback test
* ci: apply automated fixes
* fix: stabilize persisted cursor paging and node conformance harness
* test: add shared conformance suite for electron persisted
* ci: apply automated fixes
* fix: harden electron full-e2e transport and timing
* ci: apply automated fixes
* fix: preserve bigint payloads across electron fixture bridge
* ci: apply automated fixes
* fix: serialize electron e2e results and extend full-mode timeouts
* fix: raise electron full-mode test time budgets via vitest config
* ci: apply automated fixes
* fix: use function-safe e2e input encoding for electron bridge
* fix: strip subscription handles from electron loadSubset rpc
* fix: preserve shared ir nodes in electron e2e input encoder
* fix: make electron main require base path cjs-compatible
* test: serialize full-mode electron suites and align ipc timeouts
* ci: apply automated fixes
* test: isolate electron fixture env from vitest coverage vars
* test: increase full-mode electron rpc time budgets
* fix: resolve electron require base path from absolute argv entry
* feat: add react-native/expo sqlite persisted collection package
* ci: apply automated fixes
* fix: align mobile adapter typing and burst persistence test
* ci: apply automated fixes
* fix(react-native): harden transactions and expand parity coverage
* ci: apply automated fixes
* test(react-native): stabilize lifecycle and runtime contract coverage
* ci: apply automated fixes
* fix(react-native): resolve transaction deadlock and harden runtime test rails
* ci: apply automated fixes
* fix(sqlite-driver): enforce transaction callback parity across runtimes
* ci: apply automated fixes
* test(react-native): add expo lifecycle coverage and ci e2e wiring
* docs(ci): fix mobile setup examples and enforce runtime lane
* ci: apply automated fixes
* ci(e2e): make mobile runtime lane enforceable but non-blocking by default
* ci(e2e): run node and electron persisted suites
* feat: add cloudflare DO sqlite persistence with wrangler e2e
* test: harden cloudflare DO schema and transaction semantics
* docs: add complete API readmes for sqlite persistence packages
* refactor: unify sqlite persistence APIs around shared persisters
* fix: preserve adapter defaults while keeping mode-aware persistence
* refactor: simplify sqlite persistence APIs across runtimes
* fix: satisfy generic cast in node persistence
* fix: narrow generic adapter cast in mobile persistence
* fix: narrow generic adapter cast in cloudflare persistence
* fix: tighten electron test harness and renderer typing
* fix: tighten electron ipc test persistence typing
* fix: normalize markIndexRemoved return type in electron tests
* fix(cloudflare-e2e): use collection-resolved adapter in worker fixture
* fix(electron): preserve remote error code in renderer IPC
* fix(electron): catch async adapter errors in main IPC bridge
* feat(sqlite): accept native runtime handles in persistence factories
* refactor(persistence): require native sqlite handles only
* chore: refresh pnpm lockfile for cloudflare package peers
* ci: apply automated fixes
* feat(browser): implement phase 7 single-tab persistence
* ci: apply automated fixes
* fix(browser): move phase 7 opfs path into worker
* ci: apply automated fixes
* fix(persistence): restore stream position on startup to prevent duplicate tx skipping
The PersistedCollectionRuntime never restored its stream position from
the database on startup, always beginning at localTerm=1, localSeq=0.
After a page reload, the first new mutation would collide with a
previously applied transaction (term=1, seq=1), causing the SQLite
adapter's applyCommittedTx to silently skip it as a duplicate.
Add getStreamPosition to PersistenceAdapter (optional) and implement it
in SQLiteCorePersistenceAdapter. Call it from startInternal() so that
observeStreamPosition seeds the local counters before any mutations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(persistence): inline row data in tx:committed broadcasts
Include full row values in the tx:committed message so receiving tabs
can apply changes directly without a SQLite round-trip via loadRowsByKeys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: apply automated fixes
* ci: apply automated fixes (attempt 2/3)
* docs: update phase 8 plan with implementation status annotations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: apply automated fixes
* feat(browser): implement Phase 8 BrowserCollectionCoordinator for multi-tab support
Web Locks for per-collection leadership election, BroadcastChannel for
cross-tab RPC transport, DB writer lock for SQLite write serialization,
envelope dedup for exactly-once mutations, and leader heartbeats.
Includes 15 unit tests with Web Locks/BroadcastChannel mocks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: apply automated fixes
* fix(browser): prevent stuck leadership state on acquireLeadership errors
Previously, `state.isLeader = true` was set before the setup code that
calls `getStreamPosition()`. If `getStreamPosition` threw (e.g. due to
a UNIQUE constraint violation from React StrictMode double-mounting),
`isLeader` remained permanently stuck at `true` because the `finally`
block that resets it was inside an inner try/finally that was never
entered.
Fix: Wrap the entire lock callback body in a single try/finally. Set
`state.isLeader = true` only after successful setup (stream position
restore and term increment). The finally block always runs and resets
`isLeader = false` + cleans up the heartbeat timer.
Also refactors the coordinator to support lazy adapter wiring via
`setAdapter()`, allowing `createBrowserWASQLitePersistence` to inject
the adapter after construction. This enables the demo to construct the
coordinator without requiring the adapter upfront.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(persistence): always route mutations through coordinator to prevent seq collisions
The leader tab had two mutation paths: a "direct" path (write to SQLite
and broadcast) and an RPC path (through the coordinator). Previously,
only follower tabs used the RPC path — the leader bypassed the
coordinator and wrote directly.
This caused a seq collision: the leader's direct writes incremented the
runtime's `localSeq` but left the coordinator's `state.latestSeq` at 0.
When a follower later sent an RPC, the coordinator assigned seq starting
from 1 again, producing duplicate seq numbers. The leader then skipped
these "already-seen" tx:committed messages, causing follower mutations
to silently disappear.
Fix: Always route through `requestApplyLocalMutations` when available,
regardless of leader/follower status. This keeps the coordinator's seq
counter in sync with all writes.
Also removes `requestApplyLocalMutations` from `SingleProcessCoordinator`
— it was a stub that returned success without persisting, which would
break now that the leader uses this path. Single-process mode correctly
falls back to the direct path since it has no multi-tab coordination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(persistence): allow leader to process coordinator-delivered tx:committed messages
The leader tab's `onCoordinatorMessage` handler skipped ALL messages
where `senderId` matched the coordinator's own node ID. But when the
coordinator processes a follower's RPC in `handleApplyLocalMutations`,
it delivers the resulting `tx:committed` to local subscribers using the
coordinator's own `senderId`. This caused the leader's runtime to
silently ignore follower mutations — they were written to SQLite but
never applied to the leader's in-memory collection.
Fix: Allow `tx:committed` messages from self to pass through the filter.
The seq dedup logic in `processCommittedTxUnsafe` already prevents
double-processing: when the leader's own mutations go through the
coordinator, `observeStreamPosition` is called with the response's
term/seq before the local `tx:committed` delivery runs under the mutex,
so the duplicate is detected via `txCommitted.seq <= this.latestSeq`.
Other message types (heartbeats, resets) from self are still skipped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: apply automated fixes
* fix(persistence): handle delete messages with value in normalizeSyncWriteMessage
When queryCollectionOptions detects a server-side deletion, it sends
{ type: 'delete', value: oldItem } through the sync. The persistence
layer only checked for 'key' in message to detect deletes, causing
value-based deletes to be misclassified as updates. Also use optional
chaining for process.versions in React Native where process exists but
versions may not.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(electron-persistence): add ElectronCollectionCoordinator for cross-window sync
Add ElectronCollectionCoordinator using BroadcastChannel + Web Locks for
leader election and cross-window coordination in Electron renderer windows.
Wire coordinator into renderer persistence via setAdapter(), add
getStreamPosition to the IPC protocol, and export from package index.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: apply automated fixes
* chore: extract electron, node, and cloudflare persistence packages to follow-up branches
Remove db-electron-sqlite-persisted-collection, db-node-sqlite-persisted-collection,
and db-cloudflare-do-sqlite-persisted-collection from this branch so it contains only
the core persistence packages (db, db-sqlite-persisted-collection-core,
db-browser-wa-sqlite-persisted-collection, db-react-native-sqlite-persisted-collection).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ci): remove extra blank line in e2e-tests workflow
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Delete plan
* Changeset
* Strip virtual props from persistence tests
* Changeset fix
* Formatting
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>1 parent 43ecbfa commit d351c67
73 files changed
Lines changed: 14128 additions & 25 deletions
File tree
- .changeset
- .github/workflows
- packages
- db-browser-wa-sqlite-persisted-collection
- e2e
- src
- tests
- helpers
- db-react-native-sqlite-persisted-collection
- e2e
- src
- tests
- helpers
- db-sqlite-persisted-collection-core
- src
- tests
- contracts
- db
- src
- collection
- indexes
- query
- compiler
- tests
- query/compiler
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
65 | 102 | | |
66 | 103 | | |
67 | 104 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
14 | 18 | | |
15 | 19 | | |
16 | 20 | | |
| |||
Lines changed: 54 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
0 commit comments