perf: sub-100ms incremental rebuilds (466ms → 67-80ms) (#644)

carlos-alm · web-flow · commit f6bd74eb29fd · 2026-03-26T22:08:14.000-06:00
* fix: db version warning, barrel export tracing, quieter tsconfig, Set compat 9.1 — Warn on graph load when DB was built with a different codegraph version. The check runs once per process in openReadonlyOrFail() and suggests `build --no-incremental`. 9.2 — Barrel-only files now emit reexport edges during build. Previously the entire file was skipped in buildImportEdges; now only non-reexport imports are skipped, so `codegraph exports` can follow re-export chains. 9.3 — Demote "Failed to parse tsconfig.json" from warn to debug level so it no longer clutters every build output. 9.4 — Document EXTENSIONS/IGNORE_DIRS Array→Set breaking change in CHANGELOG. Add .toArray() convenience method and export ArrayCompatSet type for consumers migrating from the pre-3.4 array API. * docs: soften EXTENSIONS/IGNORE_DIRS changelog wording * fix: address review feedback — version check, Set mutation, barrel edge duplication (#634) - Move _versionWarned flag outside mismatch conditional to avoid redundant build_meta queries when versions match. - Wrap SUPPORTED_EXTENSIONS in new Set() to avoid mutating the sibling module's export. - Delete outgoing edges for barrel-only files before re-adding them to fileSymbols during incremental builds, preventing duplicate reexport edges. * refactor: replace vendor.d.ts with @types/better-sqlite3 Delete the 39-LOC manual ambient type declarations for better-sqlite3 and use the community @types/better-sqlite3 package instead. The vendor file was a migration-era shim (allowJs is long gone from tsconfig). - Replace all BetterSqlite3.Database → BetterSqlite3Database (types.ts) - Replace all BetterSqlite3.Statement → SqliteStatement (types.ts) - Simplify constructor casts in connection.ts, branch-compare.ts, snapshot.ts (no longer needed with proper @types) - Clean up watcher.ts double-cast and info.ts @ts-expect-error - Widen transaction() return type for @types compatibility * fix: address Greptile review feedback - Restore warn level for tsconfig/jsconfig parse errors (P1: was incorrectly downgraded to debug; ENOENT is already guarded by existsSync before the try block) - Simplify openReadonlyOrFail constructor cast to match openDb pattern (P2) - Use Object.assign in withArrayCompat instead of cast-then-mutate (P2) - Remove unused BetterSqlite3Database import from branch-compare.ts - Remove stale biome-ignore suppression from snapshot.ts * fix: preserve transaction argument types via inline inference (#640) * perf: sub-100ms 1-file incremental rebuilds (466ms → 78-90ms) Four optimizations for small incremental builds (≤5 changed files): 1. Scope barrel re-parsing to related barrels only (resolve-imports.ts) Instead of parsing ALL barrel files one-by-one (~93ms), only re-parse barrels imported by or re-exporting from changed files, batch-parsed in one call (~11ms). 2. Fast-path structure metrics (build-structure.ts) For ≤5 changed files on large codebases (>20 files), use targeted per-file SQL queries (~2ms) instead of loading ALL definitions from DB and recomputing ALL metrics (~35ms). 3. Skip unnecessary finalize work (finalize.ts) - Skip setBuildMeta writes for ≤5 files (avoids WAL transaction) - Skip drift detection for ≤3 files - Skip auto-registration dynamic import for incremental builds - Move timing measurement before db.close() 4. Deferred db.close() for small incremental builds (connection.ts) WAL checkpoint in db.close() costs ~250ms on Windows NTFS. Defer to next event loop tick so buildGraph() returns immediately. Includes flushDeferredClose() for test compatibility and auto-flush on openDb(). * perf: scope node loading and skip filesystem scan for incremental builds Two optimizations for small incremental builds (≤5 changed files): 1. collectFiles: reconstruct file list from DB file_hashes + journal deltas instead of full recursive filesystem scan (~7ms savings) 2. buildEdges: scope node loading query to only relevant files (changed files + import targets) with lazy SQL fallback for global name lookups (~5ms savings on 6K→400 nodes) Combined improvement: 78-90ms → 67-80ms for 1-file incremental on 473-file codebase. * fix: align scopedLoad gate with loadNodes to prevent missing lazy fallback (#644) Return scoped flag directly from loadNodes instead of re-deriving it with a different threshold, ensuring addLazyFallback is always called when the query was actually scoped. * fix: align drift detection and metadata persistence thresholds, remove tmpdir shadow (#644) - Lower setBuildMeta gate from >5 to >3 to match drift detection gate, preventing stale baseline counts after series of 4-5 file builds. - Remove redundant dynamic import of tmpdir (already imported statically). - Add comment clarifying finalizeMs metric placement.
diff --git a/src/domain/graph/builder/stages/build-edges.ts b/src/domain/graph/builder/stages/build-edges.ts
@@ -561,17 +561,82 @@ function buildClassHierarchyEdges(
 
 // ── Main entry point ────────────────────────────────────────────────────
 
+/**
+ * For small incremental builds (≤5 changed files on a large codebase), scope
+ * the node loading query to only files that are relevant: changed files +
+ * their import targets. Falls back to loading ALL nodes for full builds or
+ * larger incremental changes.
+ */
+function loadNodes(ctx: PipelineContext): { rows: QueryNodeRow[]; scoped: boolean } {
+  const { db, fileSymbols, isFullBuild, batchResolved } = ctx;
+  const nodeKindFilter = `kind IN ('function','method','class','interface','struct','type','module','enum','trait','record','constant')`;
+
+  // Gate: only scope for small incremental on large codebases
+  if (!isFullBuild && fileSymbols.size <= 5) {
+    const existingFileCount = (
+      db.prepare("SELECT COUNT(*) as c FROM nodes WHERE kind = 'file'").get() as { c: number }
+    ).c;
+    if (existingFileCount > 20) {
+      // Collect relevant files: changed files + their import targets
+      const relevantFiles = new Set<string>(fileSymbols.keys());
+      if (batchResolved) {
+        for (const resolvedPath of batchResolved.values()) {
+          relevantFiles.add(resolvedPath);
+        }
+      }
+      // Also add barrel-only files
+      for (const barrelPath of ctx.barrelOnlyFiles) {
+        relevantFiles.add(barrelPath);
+      }
+
+      const placeholders = [...relevantFiles].map(() => '?').join(',');
+      const rows = db
+        .prepare(
+          `SELECT id, name, kind, file, line FROM nodes WHERE ${nodeKindFilter} AND file IN (${placeholders})`,
+        )
+        .all(...relevantFiles) as QueryNodeRow[];
+      return { rows, scoped: true };
+    }
+  }
+
+  const rows = db
+    .prepare(`SELECT id, name, kind, file, line FROM nodes WHERE ${nodeKindFilter}`)
+    .all() as QueryNodeRow[];
+  return { rows, scoped: false };
+}
+
+/**
+ * For scoped node loading, patch nodesByName.get with a lazy SQL fallback
+ * so global name-only lookups (resolveByMethodOrGlobal, supplementReceiverEdges)
+ * can still find nodes outside the scoped set.
+ */
+function addLazyFallback(ctx: PipelineContext, scopedLoad: boolean): void {
+  if (!scopedLoad) return;
+  const { db } = ctx;
+  const fallbackStmt = db.prepare(
+    `SELECT id, name, kind, file, line FROM nodes WHERE name = ? AND kind != 'file'`,
+  );
+  const originalGet = ctx.nodesByName.get.bind(ctx.nodesByName);
+  ctx.nodesByName.get = (name: string) => {
+    const result = originalGet(name);
+    if (result !== undefined) return result;
+    const rows = fallbackStmt.all(name) as unknown as NodeRow[];
+    if (rows.length > 0) {
+      ctx.nodesByName.set(name, rows);
+      return rows;
+    }
+    return undefined;
+  };
+}
+
 export async function buildEdges(ctx: PipelineContext): Promise<void> {
   const { db, engineName } = ctx;
 
   const getNodeIdStmt = makeGetNodeIdStmt(db);
 
-  const allNodes = db
-    .prepare(
-      `SELECT id, name, kind, file, line FROM nodes WHERE kind IN ('function','method','class','interface','struct','type','module','enum','trait','record','constant')`,
-    )
-    .all() as QueryNodeRow[];
-  setupNodeLookups(ctx, allNodes);
+  const { rows: allNodesBefore, scoped: scopedLoad } = loadNodes(ctx);
+  setupNodeLookups(ctx, allNodesBefore);
+  addLazyFallback(ctx, scopedLoad);
 
   const t0 = performance.now();
   const buildEdgesTx = db.transaction(() => {
@@ -592,7 +657,7 @@ export async function buildEdges(ctx: PipelineContext): Promise<void> {
 
     const native = engineName === 'native' ? loadNative() : null;
     if (native?.buildCallEdges) {
-      buildCallEdgesNative(ctx, getNodeIdStmt, allEdgeRows, allNodes, native);
+      buildCallEdgesNative(ctx, getNodeIdStmt, allEdgeRows, allNodesBefore, native);
     } else {
       buildCallEdgesJS(ctx, getNodeIdStmt, allEdgeRows);
     }
diff --git a/src/domain/graph/builder/stages/collect-files.ts b/src/domain/graph/builder/stages/collect-files.ts
@@ -2,14 +2,78 @@
  * Stage: collectFiles
  *
  * Collects all source files to process. Handles both normal and scoped rebuilds.
+ * For incremental builds with a valid journal, reconstructs the file list from
+ * the DB's file_hashes table + journal deltas, skipping the filesystem scan.
  */
 import fs from 'node:fs';
 import path from 'node:path';
-import { info } from '../../../../infrastructure/logger.js';
+import { debug, info } from '../../../../infrastructure/logger.js';
 import { normalizePath } from '../../../../shared/constants.js';
+import { readJournal } from '../../journal.js';
 import type { PipelineContext } from '../context.js';
 import { collectFiles as collectFilesUtil } from '../helpers.js';
 
+/**
+ * Reconstruct allFiles from DB file_hashes + journal deltas.
+ * Returns null when the fast path isn't applicable (first build, no journal, etc).
+ */
+function tryFastCollect(
+  ctx: PipelineContext,
+): { files: string[]; directories: Set<string> } | null {
+  const { db, rootDir } = ctx;
+
+  // 1. Check that file_hashes table exists and has entries
+  let dbFileCount: number;
+  try {
+    dbFileCount = (db.prepare('SELECT COUNT(*) as c FROM file_hashes').get() as { c: number }).c;
+  } catch {
+    return null;
+  }
+  if (dbFileCount === 0) return null;
+
+  // 2. Read the journal — only use fast path when journal has entries,
+  // proving the watcher was active and tracking changes. An empty-but-valid
+  // journal (no watcher) could miss file deletions.
+  const journal = readJournal(rootDir);
+  if (!journal.valid) return null;
+  const hasEntries =
+    (journal.changed && journal.changed.length > 0) ||
+    (journal.removed && journal.removed.length > 0);
+  if (!hasEntries) return null;
+
+  // 3. Load existing file list from file_hashes (relative paths)
+  const dbFiles = (db.prepare('SELECT file FROM file_hashes').all() as Array<{ file: string }>).map(
+    (r) => r.file,
+  );
+
+  // 4. Apply journal deltas: remove deleted files, add new/changed files
+  const fileSet = new Set(dbFiles);
+  if (journal.removed) {
+    for (const removed of journal.removed) {
+      fileSet.delete(removed);
+    }
+  }
+  if (journal.changed) {
+    for (const changed of journal.changed) {
+      fileSet.add(changed);
+    }
+  }
+
+  // 5. Convert to absolute paths and compute directories
+  const files: string[] = [];
+  const directories = new Set<string>();
+  for (const relPath of fileSet) {
+    const absPath = path.join(rootDir, relPath);
+    files.push(absPath);
+    directories.add(path.dirname(absPath));
+  }
+
+  debug(
+    `collectFiles fast path: ${dbFiles.length} from DB, journal: +${journal.changed?.length ?? 0}/-${journal.removed?.length ?? 0} → ${files.length} files`,
+  );
+  return { files, directories };
+}
+
 export async function collectFiles(ctx: PipelineContext): Promise<void> {
   const { rootDir, config, opts } = ctx;
 
@@ -33,10 +97,23 @@ export async function collectFiles(ctx: PipelineContext): Promise<void> {
     ctx.removed = missing;
     ctx.isFullBuild = false;
     info(`Scoped rebuild: ${existing.length} files to rebuild, ${missing.length} to purge`);
-  } else {
-    const collected = collectFilesUtil(rootDir, [], config, new Set<string>());
-    ctx.allFiles = collected.files;
-    ctx.discoveredDirs = collected.directories;
-    info(`Found ${ctx.allFiles.length} files to parse`);
+    return;
   }
+
+  // Incremental fast path: reconstruct file list from DB + journal deltas
+  // instead of full recursive filesystem scan (~8ms savings on 473 files).
+  if (ctx.incremental && !ctx.forceFullRebuild) {
+    const fast = tryFastCollect(ctx);
+    if (fast) {
+      ctx.allFiles = fast.files;
+      ctx.discoveredDirs = fast.directories;
+      info(`Found ${ctx.allFiles.length} files (cached)`);
+      return;
+    }
+  }
+
+  const collected = collectFilesUtil(rootDir, [], config, new Set<string>());
+  ctx.allFiles = collected.files;
+  ctx.discoveredDirs = collected.directories;
+  info(`Found ${ctx.allFiles.length} files to parse`);
 }
diff --git a/src/domain/graph/builder/stages/finalize.ts b/src/domain/graph/builder/stages/finalize.ts
@@ -67,7 +67,9 @@ export async function finalize(ctx: PipelineContext): Promise<void> {
   // built_at is only used by stale-embeddings check (skipped for incremental),
   // and counts are only used by drift detection (skipped for ≤3 files).
   // This avoids a transaction commit + WAL fsync (~15-30ms).
-  if (isFullBuild || allSymbols.size > 5) {
+  // Threshold aligned with drift detection gate (allSymbols.size > 3) so stored
+  // counts stay fresh whenever drift detection reads them.
+  if (isFullBuild || allSymbols.size > 3) {
     try {
       setBuildMeta(db, {
         engine: ctx.engineName,
@@ -157,6 +159,10 @@ export async function finalize(ctx: PipelineContext): Promise<void> {
     }
   }
 
+  // Intentionally measured before closeDb / writeJournalHeader / auto-registration:
+  // for the deferred-close path the close is async (setImmediate), and for full
+  // builds the metric captures finalize logic only — DB close cost is tracked
+  // separately via timing.closeDbMs when available.
   ctx.timing.finalizeMs = performance.now() - t0;
 
   // For small incremental builds, defer db.close() to the next event loop tick.
@@ -177,7 +183,6 @@ export async function finalize(ctx: PipelineContext): Promise<void> {
   // registered during the initial full build. The dynamic import + file I/O
   // costs ~100ms which dominates incremental finalize time.
   if (!opts.skipRegistry && isFullBuild) {
-    const { tmpdir } = await import('node:os');
     const tmpDir = path.resolve(tmpdir());
     const resolvedRoot = path.resolve(rootDir);
     if (resolvedRoot.startsWith(tmpDir)) {
