You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(roadmap): plan full rusqlite DB migration (Phase 6.13–6.17) (#652)
* docs(roadmap): add Phase 6.13–6.17 for full rusqlite DB migration
Plan the complete migration of all DB operations to rusqlite on the
native engine path, so better-sqlite3 is only used for WASM fallback:
- 6.13: NativeDatabase class (connection lifecycle)
- 6.14: Native read queries (41 Repository methods)
- 6.15: Native write operations (build pipeline)
- 6.16: Dynamic SQL & edge cases
- 6.17: Cleanup & better-sqlite3 isolation
* fix(roadmap): correct dep count claim in 6.17 — rusqlite is compiled-in, not an npm dep (#652)
|[**6**](#phase-6--native-analysis-acceleration)| Native Analysis Acceleration | Rust extraction for AST/CFG/dataflow/complexity; batch SQLite inserts; incremental rebuilds; native DB write pipeline to close the WASM parity gap on full builds|**In Progress** (7 of 12 done, 1 partial) |
22
+
|[**6**](#phase-6--native-analysis-acceleration)| Native Analysis Acceleration | Rust extraction for AST/CFG/dataflow/complexity; batch SQLite inserts; incremental rebuilds; native DB write pipeline; full rusqlite migration so native engine never touches better-sqlite3|**In Progress** (7 of 17 done, 1 partial) |
23
23
|[**7**](#phase-7--expanded-language-support)| Expanded Language Support | Parser abstraction layer, 23 new languages in 4 batches (11 → 34), dual-engine support | Planned |
### 6.13 -- NativeDatabase Class (rusqlite Connection Lifecycle)
1282
+
1283
+
**Not started.** Foundation for moving all DB operations to `rusqlite` on the native engine path. Currently `better-sqlite3` (JS) handles all DB operations for both engines, and `rusqlite` is only used for bulk AST node insertion (6.9/PR #651). The goal is: **native engine → rusqlite for all DB; WASM engine → better-sqlite3 for all DB** — eliminating the dual-SQLite-in-one-process problem and unlocking Rust-speed for every query.
1284
+
1285
+
**Plan:**
1286
+
-**Create `NativeDatabase` napi-rs class** in `crates/codegraph-core/src/native_db.rs` holding a `rusqlite::Connection`
**Not started.** Migrate all 41 `Repository` read methods to Rust, so every query runs via `rusqlite` on the native engine. The existing `Repository` abstract class and `SqliteRepository` provide the exact seam — each method is a fixed SQL query with typed parameters and results.
1298
+
1299
+
**Plan:**
1300
+
-**Implement each Repository method as a Rust method on `NativeDatabase`:** Start with simple ones (`countNodes`, `countEdges`, `countFiles`, `findNodeById`), then fixed-SQL edge queries (16 methods), then parameterized queries with dynamic filtering
1301
+
-**Replicate `NodeQuery` fluent builder in Rust:** The dynamic SQL builder used by `findNodesWithFanIn`, `findNodesForTriage`, `listFunctionNodes` must produce identical SQL and results
1302
+
-**Create `NativeRepository extends Repository`** in `src/db/repository/native-repository.ts` — delegates all 41 methods to `NativeDatabase` napi calls
1303
+
-**Wire `openRepo()` to return `NativeRepository`** when native engine is available
1304
+
-**Parity test suite:** Run every Repository method on both `SqliteRepository` and `NativeRepository` against the same DB, assert identical output
**Not started.** Migrate all build-pipeline write operations to `rusqlite`, so the entire build (parse → insert → finalize) uses a single Rust-side DB connection on native. This consolidates the scattered rusqlite usage from 6.9–6.12 into the `NativeDatabase` class and adds the remaining write paths.
1311
+
1312
+
**Plan:**
1313
+
-**Migrate `batchInsertNodes` and `batchInsertEdges`** — high-value; currently the hottest build path after parse
1314
+
-**Migrate `purgeFilesData`** — cascade DELETE across 10 tables during incremental rebuilds
1315
+
-**Migrate complexity/CFG/dataflow/co-change writes** — consolidate the per-phase Rust inserts from 6.9/6.10 into `NativeDatabase` methods
1316
+
-**Migrate `upsertFileHashes` and `updateExportedFlags`** — finalize-phase operations
1317
+
-**Consolidate `bulk_insert_ast_nodes`** into `NativeDatabase` (currently opens its own separate connection)
1318
+
-**Update `PipelineContext`** to thread `NativeDatabase` through all build stages when native engine is active
1319
+
-**Transactional parity testing:** Verify that partial failures, rollbacks, and WAL behavior are identical between engines
**Not started.** Handle the remaining non-trivial DB patterns that don't map cleanly to fixed Repository methods.
1326
+
1327
+
**Plan:**
1328
+
-**`NodeQuery` builder edge cases:** Ensure the Rust-side replica handles all filter combinations, JOIN paths, ORDER BY variations, and LIMIT/OFFSET correctly — fuzz-test with random filter combinations against the JS builder
1329
+
-**`openReadonlyOrFail` version-check logic:** Port the schema-version validation that runs on read-only DB opens
1330
+
-**Advisory lock mechanism:** Keep in JS (filesystem-based, not SQLite) — ensure `NativeDatabase.close()` integrates with the existing lock lifecycle
1331
+
-**`closeDbDeferred` / WAL checkpoint deferral:** Keep deferred-close logic in JS, call `NativeDatabase.close()` when ready
1332
+
-**Raw `db.prepare()` stragglers:** Audit all 383 callers of `.prepare()` and ensure every one routes through either `Repository` or `NativeDatabase` methods — no direct better-sqlite3 usage on the native path
**Not started.** Final step: ensure `better-sqlite3` is only loaded when running the WASM engine. Update documentation and dependency claims.
1339
+
1340
+
**Plan:**
1341
+
-**Lazy-load `better-sqlite3`** — only `require()` it when engine is WASM; native path never touches it
1342
+
-**Remove the double-connection pattern** — `bulk_insert_ast_nodes` (6.9) and any other Rust functions that open their own `rusqlite::Connection` should use the shared `NativeDatabase` instance
1343
+
-**Profile and tune:** Enable `rusqlite` statement caching, optimize batch sizes, tune WAL settings for the unified Rust connection
1344
+
-**Update README.md** — clarify that `better-sqlite3` is now WASM-engine only (the "3 runtime dependencies" count stays the same because `rusqlite` is compiled into the native addon, not a separate npm dependency)
1345
+
-**Update verification strategy:** Add a parity gate that runs the full test suite on both engines and diffs the resulting DBs row-by-row
0 commit comments