Product Spike: User feedback loops + community templates

[anand-testcompare]

Why

We want a product-level feedback loop where users can generate a first-draft diagram, edit it in an embedded Excalidraw UI, and those edits/deltas become the gold standard for improving future generations.

This spike also explores a community angle: sharing optimized/stylized templates (similar to Vercel-style "skills") that other users can reuse.

Problem Statement

Given a user-provided input (e.g. pasted text or a long SME interview transcript):

1. Backend generates an initial diagram.
2. User edits the diagram directly in the Excalidraw canvas.
3. The system captures the deltas between generated output and final user-approved diagram.
4. Those deltas can later be used to improve template defaults and/or generation quality for similar future inputs.

Questions to Answer (No Implementation)

UX + workflow

• What is the intended editing experience (embedded Excalidraw iframe vs in-app canvas component)?
• What is the "save" unit: per-diagram, per-template, per-organization, per-scenario?
• What does it mean for a user to be "satisfied" (explicit approval button, publish, version tag)?

Data capture

• What exactly is the "delta" we store?
  • Full before/after elements?
  • Patch/diff (Excalidraw element diff)?
  • Event log of edits?
• What metadata is required to make the delta useful?
  • prompt / intermediate
  • diagramType + templateId + templateVersion
  • model + settings
  • timestamps + user/org

Storage + source of truth

• If templates eventually live in Convex:
  • how do we represent "base template" vs "user variant"?
  • what does versioning look like?
• What are privacy/security constraints for storing user diagrams/deltas (esp. SME interview content)?

Optimization loop ("reverse dspy")

• What is the minimal feasible optimization approach:
  • manual curation -> update template defaults
  • automatic mining of common edit patterns -> propose changes
  • few-shot examples per diagramType/template variant
• How do we measure improvement and avoid regressions?

Community templates

• What is shareable?
  • style-only templates vs semantic templates
  • per-diagram-type constraints
• How do we prevent a low-quality template marketplace?
  • moderation, rating, verification, preview + grader

Deliverables

• A short design doc (1-2 pages) describing the workflow and the data model.
• Proposed Convex tables/fields (rough), including a versioning strategy.
• Clear recommendation: what to build first, what to defer.

Out of Scope

• Building the UI.
• Implementing the optimization loop.
• Shipping a template marketplace.