feat(redact): add OpenAI Privacy Filter as optional 8th detection layer

[peyton-alt]

Summary

Opt-in 8th redaction layer that shells out to the user-installed OpenAI Privacy Filter (opf) binary. Runs only at condensation + export boundaries; per-turn writes stay on the existing 7-layer pipeline. Default-off; users opt in via redaction.openai_privacy_filter.enabled = true.

This PR was rewritten from main on 2026-05-14. The first iteration accumulated review-fix scope across multiple bot review passes and reached ~3000 insertions. The rewrite keeps every real correctness/privacy fix found during that cycle while skipping speculative scope (~1547 insertions, 4 commits, 19 files). The original work is preserved at feat/openai-privacy-filter-v1-backup for reference.

What's included

• Two public entry points (StringWithPrivacyFilter, JSONLContentWithPrivacyFilter) plus thin Bytes wrappers — four plain entry points unchanged so per-turn temp writes never invoke OPF
• \x1e-joined batched inference: one shell-out per scope amortizes the OPF cold-start across the whole transcript
• Process-scoped circuit breaker: first OPF failure disables it for the rest of the invocation (closes OPF: process-scoped circuit breaker after first failure #1218)
• Split createRedactedBlobFromFile so per-turn temp writes (addDirectoryToChanges) stay on the plain pipeline while copyMetadataDir uses the OPF-enabled variant
• PromptsRedactedContent field on Write/UpdateCommittedOptions: pre-compute the joined-prompt redaction once in finalizeAllTurnCheckpoints and condenseSingleCheckpoint so multi-checkpoint commits don't re-run OPF N×
• Settings layer rejects unknown category names at parse time — silent zero-detection of a privacy category is effectively a correctness bug
• Sanitized stdout/stderr in shellout errors (OPF could echo input content; bytes-only diagnostics)
• charToByteOffset correct for multibyte UTF-8 (no off-by-one at end)
• Span docs reflect byte offsets (shellout adapter converts from OPF's character offsets)
• Inline shell-out runtime, label map, and progress UX in redact/opf.go — no redact/opf_runtime/ subpackage, no separate progress writer

What's intentionally NOT included (deferred to follow-ups)

• Summary-field batching via a StringsWithPrivacyFilter API — summaries are opt-in (only when IsSummarizeEnabled), have 5–10 short fields, and add ~10s worst case. Acceptable until someone complains.
• doctor_bundle per-entry batching — diagnostic command, runs rarely, worst case slow not broken.
• on_failure settings field — dropped entirely. DisallowUnknownFields rejects any user who tries to set it. Warn is the only supported mode today; if block-mode runtime wiring lands later, the field comes back in lockstep.

Test plan

• mise run check (fmt + lint + test:ci + Vogon canary + roger-roger external-agent canary) — all green on 672044c2f
• Real-agent smoke test on a fresh repo with private_person enabled — confirmed:
  • No OpenAI Privacy Filter: scanning transcript during the agent turn (per-turn writes don't invoke OPF)
  • → scanning transcript… ✓ done (29.4s) then (2.3s) at commit time (transcript + joined-prompts passes)
  • [REDACTED_PERSON] appears in entire checkpoint explain HEAD for both the prompt and the assistant transcript

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (2)

redact/opf_runtime/shellout.go:199

• charToByteOffset accepts an offset that is one rune past the end: for a 3-rune string, charOff == 4 reaches byteOff == len(s) and returns len(s) instead of -1. That lets malformed OPF spans at EOF be treated as valid and can over-redact the last input in a batch; only charOff == runeCount should map to len(s).

	for i := range charOff {
		if byteOff >= len(s) {
			if i == charOff-1 && byteOff == len(s) {
				return byteOff
			}

cmd/entire/cli/checkpoint/committed.go:1778

• createRedactedBlobFromFile is shared by the temporary shadow-branch metadata writers (temporary.go:987 and temporary.go:1046). Switching this shared helper to the OPF-enabled redactors makes per-turn temporary metadata writes invoke OPF, contrary to the new design that keeps OPF at condensation/export boundaries and potentially adding OPF latency to every turn. Split the helper or pass a redaction mode so temporary paths continue using the plain redactors.

		redacted, jsonlErr := redact.JSONLBytesWithPrivacyFilter(ctx, content)
		if jsonlErr != nil {
			content = redact.BytesWithPrivacyFilter(ctx, content)

[peyton-alt]

@BugBot review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1012600. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 10 comments.