Web chat: support image/screenshot uploads

[andrazbajec]

Summary

The web chat interface does not support image or screenshot uploads. Users cannot paste or drag-and-drop images into the chat window.

Use case

During certification build projects (e.g. B4) and agent debugging sessions, learners want to share screenshots of error messages, terminal output, or agent responses. Currently they must copy-paste text, which works but is less convenient and doesn't capture visual context (e.g. IDE errors, browser network tabs).

Proposed behavior

• Allow users to paste images (Ctrl+V / Cmd+V) into the chat input
• Allow drag-and-drop of image files onto the chat input
• Addie (the AI) should be able to see and interpret the uploaded image

Notes

• Addie's underlying model (Claude) supports vision/image input, so the AI side should already be capable
• Main work is on the UI input and passing the image through to the model context

Labels

enhancement, web-chat, certification

[bokelley]

Triage

Classification: Feature request
Bucket(s): addie
Status: ready-for-human

What the experts said:

• prompt-engineer: Two blockers: (1) processMessage(userMessage: string) must become a multimodal content array in both the streaming and non-streaming paths — non-trivial wiring; (2) base64 image data must not be stored in addie_thread_messages.content — every subsequent turn replays the full conversation, so a single 1 MB PNG would balloon every API call for the lifetime of the thread; store [Image: filename.png] instead. Also flags the system prompt needs an explicit rule treating user-submitted images as data, not instructions, to prevent OCR-based prompt injection.
• internal-tools-strategist: Image uploads must require authentication — current endpoint is optionalAuth, and unattributed screenshots with PII (stack traces, credentials) in anonymous history is a blocker. Paste-only (no drag-drop) covers the stated use case. Inline base64 in the existing POST body (with ~1 MB size cap + isAllowedImageType allowlist enforced at the route boundary) is the right mechanism; a separate upload endpoint is overbuild.
• user-engagement-expert: Raises a conflict with BUILD_PROJECT_METHODOLOGY in certification-tools.ts (lines 104–148): Addie is explicitly policy-bound not to diagnose specific build errors — it names the category and redirects to the learner's coding assistant. A screenshot of the same error doesn't change that response, so the certification use case as written isn't served. Privacy blocker: paste/drag-drop is ambient capture — B4 learners screenshot IDEs that may have API keys or env vars in adjacent tabs; requires explicit crop + data-handling disclosure before shipping.

My take: Infrastructure exists (buildMultimodalContentBlocks, isAllowedImageType, FileReadResult) — the code path is real and bounded. But the use case premise and privacy scope need a human call first.

@bokelley, your call: Option A or Option B?

Option A — Ship paste-with-auth-gate now (requires disclosure + methodology patch):

// POST /api/addie/chat — new optional fields (rejected 401 for unauthenticated callers)
{
  "message": "Why is this failing?",
  "image_data": "iVBORw0KGgo...",
  "image_media_type": "image/png",
  "conversation_id": "..."
}

Requires: (1) auth gate, (2) size cap (~1 MB), (3) placeholder stored in history, (4) data-disclosure UI copy, (5) one-sentence extension to BUILD_PROJECT_METHODOLOGY so Addie handles image errors with the same redirect rule as text errors.

Option B — Defer: extend the certification methodology first, then ship Option A:

// certification-tools.ts — BUILD_PROJECT_METHODOLOGY addition
+ "When a learner submits an image: acknowledge the visual context in one
+  sentence, then apply the same category-only / redirect rule as for
+  text errors. Never diagnose specific errors visible in screenshots."

Same implementation as Option A, but the policy update ships first so there's no inconsistency between what learners expect and what Addie does on day one.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01SDmQBawP9KSybxW4BTgRNo

---

Generated by Claude Code