feat(scratchpad): context window management for large mcp tool outputs

[justintime4tea]

Ref: LOG-23439

Intercept large MCP tool results before they fill the context window,
save them to disk, and give the LLM eight read-only exploration tools
to selectively extract only what it needs.

Key components:

• ScratchpadWrapper: intercepts tool outputs exceeding per-tool byte
  thresholds, writes to disk, returns a compact pointer to the LLM
• Exploration tools: schema, item_schema, head, grep, get_in,
  iterate_over, slice, read — each budget-checked before returning
• ContextBudget: tracks estimated token consumption and prevents
  context overflow from scratchpad reads
• Usage tracking: records bytes_intercepted (diverted to disk) vs
  bytes_extracted (read back into context) across all workers
• aura.orchestrator.scratchpad_usage SSE event emitted at
  orchestration end with aggregate totals

Configuration:

  [mcp.servers.<name>.scratchpad]
  "*" = { min_bytes = 2000 }
  "tool_name" = { min_bytes = 4096 } # override for specific tool

  [orchestration.scratchpad]
  enabled = true
  context_safety_margin = 0.20

Ref: LOG-23533

Token Budget, Per-Call Limits and Turns

• Rename min_bytes → min_tokens with token-based interception thresholds
• Track tokens_intercepted/tokens_extracted instead of bytes throughout
• Add per-call extraction limit (max_extraction_tokens, default 10k)
• Auto-increase worker turn_depth when scratchpad active (turn_depth_bonus)
• Feed LLM-reported input_tokens back into ContextBudget via set_used_from_llm
• Replace heuristic initial_used estimates with real tokenization of MCP tool
  schemas, worker preambles, scratchpad/builtin tool definitions
• Compute per-worker worst-case using mcp_filter to count only filtered tools
• Record actual task prompt tokens in budget before worker execution

Pretty-print & minified JSON fix:

• Pretty-print JSON at write time so line-based tools work on minified responses
• WriteResult struct replaces tuple returns, includes companion file info

Companion file extraction:

• Auto-extract large structured string values from JSON as companion files
• Escaped JSON strings → pretty-printed .json companions
• Markdown content (headers + lists) → .md companions
• ContentFormat::Markdown variant with detection heuristic
• Shared prepare_write / make_companion helpers to reduce duplication

Markdown schema support:

• analyze_markdown_structure() parses ### headers into section tree with line ranges and keys
• schema tool dispatches by file extension (.json → JSON, .md → markdown sections)
• Format-aware schema_too_large error suggestions

Scratchpad tool get_in string pagination:

• offset/limit params for paginating large string values by line
• Raw string content returned (not JSON-encoded) so embedded newlines render as real lines
• Early return for non-string values, extracted get_in_paginated helper

Scratchpad tool iterate_over large string truncation:

• String values >5 lines truncated with preview + get_in path hint
• Prevents memory spikes from cloning huge strings across all array items

Companion pointer hints in wrapper:

• Pointer message lists companion files with format-specific tool suggestions

[justintime4tea]

I must have rebased and brought a couple commits in from main (the commits that aren't mine). I can dig those out but those changes are minimal and not related to scratchpad so you should be able to easily see the changes that are related to scratchpad.

[Shearerbeard]

LGTM