Switch tree-sitter wasm embed from base64 string to with { type: 'file' }

The base64-in-source approach didn't survive `bun --compile` on Windows.
The CI build's `verifyTreeSitterWasmEmbedded` step caught it:

    Embedded tree-sitter.wasm from D:\a\...\tree-sitter.wasm (205488 bytes
      → 273984 chars base64)
    [343ms]  minify  -16.58 MB
    Embedded tree-sitter wasm prefix not found in D:\a\...\codebuff.exe.

So the embed step wrote the bytes to disk and bun read them, but the
274KB string literal didn't end up in the compiled output — likely
tree-shaken or transformed by the minifier on Windows. The same code
worked on macOS and Linux locally and in CI.

Switch to Bun's documented asset-embed mechanism: import the wasm with
`with { type: 'file' }`. Bun handles this through the bundler's asset
pipeline rather than as a generic string literal, and the resulting
binary contains the wasm bytes verbatim at a bunfs path.

- cli/src/pre-init/tree-sitter-wasm.ts: import the wasm path, set the
  env var (for the locateFile fallback), and try a synchronous read so
  Parser.init can take the wasmBinary fast path. If the read throws
  (some Windows configurations have done this), log loudly so user
  reports include the diagnostic, then fall through to the locateFile
  flow — which init-node.ts now accepts bunfs paths through, even when
  fs.existsSync misreports them.
- The --smoke-tree-sitter handler is now a top-level `await` instead
  of a fire-and-forget IIFE. Without that, commander.parse() ran
  synchronously in main() and failed on the unknown flag before the
  smoke handler could exit cleanly.
- cli/scripts/build-binary.ts: drop the base64 stub-overwrite step
  entirely. New verifyTreeSitterWasmEmbedded reads a 64-byte chunk
  from the *middle* of the source wasm and asserts it appears in the
  compiled binary — that proves *this specific* tree-sitter.wasm
  shipped, not just any wasm (OpenTUI also embeds tree-sitter language
  wasms, so a magic-bytes-only scan would false-pass).
- Delete cli/src/pre-init/tree-sitter-wasm-bytes.ts: no longer used.

Verified locally: build embeds tree-sitter.wasm via the file-attribute
import, post-build verification finds the source bytes at offset
77319353 of the compiled binary, --smoke-tree-sitter exits 0 with
"tree-sitter smoke ok (wasmBinary, 205488 bytes)".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: jahooma

cli/scripts/build-binary.ts

@@ -145,11 +145,6 @@ async function main() {
   patchOpenTuiAssetPaths()
   await ensureOpenTuiNativeBundle(targetInfo)
 
-  const treeSitterEmbed = embedTreeSitterWasmAsBase64()
-  // Restore the stub even on build failure so a developer's git working
-  // tree doesn't end up with a multi-megabyte modified file.
-  process.on('exit', treeSitterEmbed.restore)
-
   const outputFilename =
     targetInfo.platform === 'win32' ? `${binaryName}.exe` : binaryName
   const outputFile = join(binDir, outputFilename)
@@ -191,20 +186,12 @@ async function main() {
 
   runCommand('bun', buildArgs, { cwd: cliRoot })
 
-  // Build done — restore the stub so a developer's working tree doesn't show
-  // a multi-megabyte diff. (The exit handler above is a backstop for crashes;
-  // the eager call here keeps a successful build clean.)
-  treeSitterEmbed.restore()
-
-  // Fail the build if the wasm bytes didn't actually make it into the
-  // compiled binary. Catches silent regressions (e.g. bun dropping a huge
-  // string literal, or some future bundler optimization) before we ship a
-  // broken artifact to users.
-  verifyTreeSitterWasmEmbedded(
-    outputFile,
-    treeSitterEmbed.wasmBase64Prefix,
-    treeSitterEmbed.wasmByteLength,
-  )
+  // Fail the build if the wasm asset didn't actually make it into the
+  // compiled binary. The pre-init imports tree-sitter.wasm with `with {
+  // type: 'file' }`, which Bun should embed; this scan catches silent
+  // regressions (e.g. tree-shaking eliminating the import) before we ship
+  // a broken artifact.
+  verifyTreeSitterWasmEmbedded(outputFile)
 
   if (targetInfo.platform !== 'win32') {
     chmodSync(outputFile, 0o755)
@@ -225,39 +212,20 @@ main().catch((error: unknown) => {
 })
 
 /**
- * Inline the contents of `web-tree-sitter/tree-sitter.wasm` as a base64 string
- * literal in `cli/src/pre-init/tree-sitter-wasm-bytes.ts`. The committed
- * file is a stub; this overwrites it with the real bytes immediately before
- * `bun build --compile`, so the bytes get baked into the binary's text
- * segment instead of being placed at a bunfs path that has to be fs-read at
- * runtime.
+ * Sanity-check the compiled binary actually contains web-tree-sitter's
+ * tree-sitter.wasm. The pre-init imports it via `with { type: 'file' }`,
+ * which should bundle the asset at a bunfs path. If tree-shaking or a
+ * future bundler change drops the import, the binary still compiles but
+ * tree-sitter init fails at runtime — this scan fails the build before
+ * we upload that artifact.
  *
- * Returns a function that restores the stub. Always invoke it (success or
- * failure) so a developer's working tree doesn't show a multi-MB diff.
+ * Looks for the actual wasm bytes (a unique 64-byte chunk pulled from
+ * the source file's interior), not just the wasm magic header — OpenTUI
+ * embeds its own tree-sitter language wasms, so a magic-bytes-only scan
+ * would false-pass even without our import. A literal bytes match
+ * proves *this specific* wasm shipped.
  */
-function embedTreeSitterWasmAsBase64(): {
-  restore: () => void
-  wasmBase64Prefix: string
-  wasmByteLength: number
-} {
-  const stubPath = join(cliRoot, 'src', 'pre-init', 'tree-sitter-wasm-bytes.ts')
-  const originalStub = readFileSync(stubPath, 'utf8')
-  let restored = false
-  const restore = (): void => {
-    if (restored) return
-    restored = true
-    try {
-      writeFileSync(stubPath, originalStub)
-    } catch (error) {
-      console.error('Failed to restore tree-sitter-wasm-bytes stub:', error)
-    }
-  }
-
-  // Try multiple candidate locations because bun's hoisting differs by
-  // platform and install command — Windows CI does `bun install --cwd cli`
-  // which can leave web-tree-sitter in cli/node_modules, while monorepo
-  // root installs hoist it to ../node_modules. Fall back to createRequire
-  // last so any failure surfaces with the full search trail.
+function verifyTreeSitterWasmEmbedded(outputFile: string): void {
   const candidates = [
     join(cliRoot, 'node_modules', 'web-tree-sitter', 'tree-sitter.wasm'),
     join(cliRoot, '..', 'node_modules', 'web-tree-sitter', 'tree-sitter.wasm'),
@@ -270,64 +238,37 @@ function embedTreeSitterWasmAsBase64(): {
       wasmPath = cliRequire.resolve('web-tree-sitter/tree-sitter.wasm')
     } catch (err) {
       throw new Error(
-        `Could not locate web-tree-sitter/tree-sitter.wasm. Searched:\n  - ` +
+        `Could not locate web-tree-sitter/tree-sitter.wasm to verify against. Searched:\n  - ` +
           candidates.join('\n  - ') +
           `\nAnd createRequire failed: ${err instanceof Error ? err.message : String(err)}`,
       )
     }
   }
 
-  const wasmBytes = readFileSync(wasmPath)
-  const base64 = wasmBytes.toString('base64')
-
-  const generated =
-    `// AUTO-GENERATED by cli/scripts/build-binary.ts during \`bun build --compile\`.\n` +
-    `// Restored to the empty stub after the build finishes — do not commit a\n` +
-    `// non-empty value here.\n` +
-    `export const TREE_SITTER_WASM_BASE64 = ${JSON.stringify(base64)}\n`
-
-  writeFileSync(stubPath, generated)
-  // Always-on log (not behind VERBOSE) so CI shows which path was used and
-  // whether the embed succeeded — this is the single most useful breadcrumb
-  // when the runtime check fails on a user machine.
-  logAlways(
-    `Embedded tree-sitter.wasm from ${wasmPath} (${wasmBytes.length} bytes → ${base64.length} chars base64)`,
-  )
-  return {
-    restore,
-    wasmBase64Prefix: base64.slice(0, 40),
-    wasmByteLength: wasmBytes.length,
-  }
-}
+  const wasm = readFileSync(wasmPath)
+  // Take a 64-byte slice from the middle of the file. The header has
+  // generic wasm magic + section markers; the tail can be padding. The
+  // middle is densely packed code/data unique to this specific wasm
+  // module.
+  const needleStart = Math.floor(wasm.length / 2)
+  const needle = wasm.subarray(needleStart, needleStart + 64)
 
-/**
- * Sanity-check the compiled binary actually contains the embedded base64.
- * If bun --compile ever silently drops a large string literal, or our embed
- * step's file write didn't take effect before the bundle ran, we want the
- * build to fail here instead of producing a binary that crashes for users.
- */
-function verifyTreeSitterWasmEmbedded(
-  outputFile: string,
-  wasmBase64Prefix: string,
-  wasmByteLength: number,
-): void {
   const binary = readFileSync(outputFile)
-  // Search as a Buffer so we don't have to load the whole binary as a UTF-8
-  // string (binaries are not valid UTF-8 and toString would corrupt bytes).
-  const needle = Buffer.from(wasmBase64Prefix, 'utf8')
   const idx = binary.indexOf(needle)
   if (idx === -1) {
     throw new Error(
-      `Embedded tree-sitter wasm prefix not found in ${outputFile}.\n` +
-        `Expected base64 prefix (first 40 chars): ${wasmBase64Prefix}\n` +
-        `Original wasm size: ${wasmByteLength} bytes.\n` +
-        `This means the build-binary.ts embed step ran but bun --compile\n` +
-        `did not include the bytes in the output. The runtime smoke test\n` +
-        `would fall back to path-based wasm resolution, which is broken on\n` +
-        `Windows.`,
+      `web-tree-sitter wasm content not found in ${outputFile}.\n` +
+        `Source wasm: ${wasmPath} (${wasm.length} bytes)\n` +
+        `Searched for 64 bytes from offset ${needleStart} of the source.\n` +
+        `Either the \`with { type: 'file' }\` import in the pre-init was\n` +
+        `tree-shaken out, or bun --compile didn't embed the asset on this\n` +
+        `platform. The runtime tree-sitter init would fail with\n` +
+        `"Internal error: tree-sitter.wasm not found".`,
     )
   }
-  logAlways(`Verified embedded wasm prefix at offset ${idx} of compiled binary.`)
+  logAlways(
+    `Verified embedded tree-sitter.wasm at offset ${idx} of compiled binary (source: ${wasmPath}).`,
+  )
 }
 
 function patchOpenTuiAssetPaths() {

cli/src/pre-init/tree-sitter-wasm-bytes.ts

@@ -1,71 +1,99 @@
 // Embed tree-sitter.wasm into the bun-compile binary so the SDK's tree-sitter
 // parser singleton can find it at runtime. Must be the very first import in
 // `index.tsx`: subsequent imports (the SDK / code-map) eagerly construct the
-// parser, and its init reads what we publish here on `globalThis`.
+// parser, and its init reads what we publish here on `globalThis` and via
+// the env var.
 //
-// Why not `with { type: 'file' }` + a runtime fs read? That's what the prior
-// fix tried, and it silently failed on Windows: bun --compile reports the
-// embedded asset path as `B:\~BUN\root\...`, and on some Windows configs
-// `fs.readFileSync` of that path throws (caught silently), so the SDK fell
-// back to path-based resolution that also failed there.
-//
-// The base64 string in `tree-sitter-wasm-bytes.ts` is replaced with the real
-// wasm contents by `cli/scripts/build-binary.ts` right before `bun build
-// --compile` and restored after. The bytes end up in the binary's text
-// segment as a JS string literal — no filesystem step on the hot path. In
-// dev / unit tests the stub is empty and code-map falls back to the
-// node_modules wasm, which works because the file actually exists locally.
+// Why `with { type: 'file' }` rather than embedding base64 in TS source:
+// the latter doesn't survive `bun --compile` on Windows. The base64 string
+// gets dropped or transformed somewhere in the bundle/minify pipeline, so
+// the runtime sees an empty stub even though the build script wrote the
+// real bytes. `with { type: 'file' }` is Bun's documented asset-embed
+// path — the file gets placed at a bunfs location the runtime can read.
+
+import { readFileSync } from 'fs'
 
-import { TREE_SITTER_WASM_BASE64 } from './tree-sitter-wasm-bytes'
+// @ts-expect-error - Bun's `with { type: 'file' }` returns a string path; TS
+// has no loader for the .wasm subpath of web-tree-sitter's package exports.
+import treeSitterWasmPath from 'web-tree-sitter/tree-sitter.wasm' with {
+  type: 'file',
+}
 
 let embeddedWasm: Uint8Array | undefined
-if (TREE_SITTER_WASM_BASE64.length > 0) {
-  const buf = Buffer.from(TREE_SITTER_WASM_BASE64, 'base64')
-  embeddedWasm = new Uint8Array(buf.buffer, buf.byteOffset, buf.byteLength)
-  // globalThis is the only cross-bundle channel: the SDK pre-built bundle
-  // inlines its own copy of `init-node.ts`, so a module-level variable in
-  // the source package isn't visible to the singleton initialized via the
-  // SDK. Slice into a fresh Uint8Array view instead of handing over the
-  // Buffer's shared underlying ArrayBuffer.
-  ;(
-    globalThis as { __CODEBUFF_TREE_SITTER_WASM_BINARY__?: Uint8Array }
-  ).__CODEBUFF_TREE_SITTER_WASM_BINARY__ = embeddedWasm
+
+if (treeSitterWasmPath) {
+  // Path stays for the locateFile fallback in init-node.ts. That fallback
+  // accepts bunfs-style paths (`/~BUN/root/...`) without checking
+  // fs.existsSync, because fs.existsSync misreports those paths on Windows.
+  // emscripten's wasm loader will fs.readFile them through its own runtime.
+  process.env.CODEBUFF_TREE_SITTER_WASM_PATH = treeSitterWasmPath
+
+  // Also try a synchronous read so we can hand the bytes straight to
+  // Parser.init via wasmBinary — bypassing locateFile entirely is the most
+  // robust path. If readFileSync of the bunfs path throws on this OS (we've
+  // seen this happen on Windows in some configurations), log it loudly so
+  // the smoke check / user reports include the diagnostic, then fall
+  // through to the locateFile flow.
+  try {
+    const buf = readFileSync(treeSitterWasmPath)
+    embeddedWasm = new Uint8Array(buf.buffer, buf.byteOffset, buf.byteLength)
+    ;(
+      globalThis as { __CODEBUFF_TREE_SITTER_WASM_BINARY__?: Uint8Array }
+    ).__CODEBUFF_TREE_SITTER_WASM_BINARY__ = embeddedWasm
+  } catch (err) {
+    console.error(
+      '[tree-sitter pre-init] readFileSync failed for embedded wasm at',
+      treeSitterWasmPath,
+      '—',
+      err instanceof Error ? err.message : String(err),
+    )
+  }
 }
 
 // Deterministic CI gate: `<binary> --smoke-tree-sitter` proves the embed
 // shipped end-to-end. Lives here, in the very first import, on purpose:
 //
 // - We're testing whether the *embed* works. Going through commander +
-//   initTreeSitterForNode would also pass via the path-resolution
-//   fallback when the embed is empty (e.g. dev mode), giving false
-//   positives that mask a broken production build.
+//   initTreeSitterForNode would pass via the path-resolution fallback
+//   when the embed is empty (e.g. dev mode), giving false positives that
+//   mask a broken production build.
 // - Failing here, before any other module loads, gives a sharp signal:
-//   the embed either worked or it didn't. No render-loop timing, no
-//   commander wiring, no SDK init order to debug.
+//   either the wasm reached the runtime or it didn't.
 //
-// Async IIFE because Parser.init returns a promise; process.exit tears
-// the process down before parallel top-level imports can fire side
-// effects we'd have to clean up.
+// Top-level await (not a fire-and-forget IIFE) because subsequent module
+// evaluation has to *wait* — otherwise `commander.parse()` runs first and
+// fails on the unknown flag before our handler can exit cleanly.
 if (process.argv.includes('--smoke-tree-sitter')) {
-  void (async () => {
-    try {
-      if (!embeddedWasm) {
-        console.error(
-          'tree-sitter smoke FAIL: TREE_SITTER_WASM_BASE64 stub is empty — ' +
-            'the build-binary.ts embed step did not run or did not write the file.',
-        )
-        process.exit(1)
-      }
-      const { Parser } = await import('web-tree-sitter')
+  try {
+    const { Parser } = await import('web-tree-sitter')
+    // Prefer the wasmBinary path (no filesystem step). Fall back to
+    // letting Parser.init resolve the path via its locateFile callback,
+    // which init-node.ts wires up to accept bunfs paths even when
+    // fs.existsSync says otherwise.
+    if (embeddedWasm) {
       await Parser.init({ wasmBinary: embeddedWasm })
-      // Marker grepped by cli/scripts/smoke-binary.ts — keep this exact text.
       console.log(
-        `tree-sitter smoke ok (${embeddedWasm.byteLength} bytes wasm initialized)`,
+        `tree-sitter smoke ok (wasmBinary, ${embeddedWasm.byteLength} bytes)`,
+      )
+    } else if (treeSitterWasmPath) {
+      await Parser.init({
+        locateFile: (name: string) =>
+          name === 'tree-sitter.wasm' ? treeSitterWasmPath : name,
+      })
+      console.log(
+        `tree-sitter smoke ok (locateFile, path=${treeSitterWasmPath})`,
+      )
+    } else {
+      console.error(
+        'tree-sitter smoke FAIL: no embedded wasm path. The `with { type: ' +
+          "'file' }` import returned a falsy value, which means the bundler " +
+          'did not embed the asset.',
       )
-      process.exit(0)
-    } catch (err) {
-      console.error('tree-sitter smoke FAIL:', err)
       process.exit(1)
     }
-  })()
+    process.exit(0)
+  } catch (err) {
+    console.error('tree-sitter smoke FAIL:', err)
+    process.exit(1)
+  }
 }