Skip to content

feat: Langfuse observability, chat forking, Prompt Lab, and split-screen comparison#1040

Open
TerminallyLazy wants to merge 1 commit intoagent0ai:developmentfrom
TerminallyLazy:a0-langfuse
Open

feat: Langfuse observability, chat forking, Prompt Lab, and split-screen comparison#1040
TerminallyLazy wants to merge 1 commit intoagent0ai:developmentfrom
TerminallyLazy:a0-langfuse

Conversation

@TerminallyLazy
Copy link
Contributor

Summary

  • Langfuse observability integration — traces every agent interaction (LLM generations, tool calls, iterations) with a built-in trace viewer for drill-down analysis
  • Chat forking — fork any conversation from any message point, creating an independent branch with full history up to that point
  • Prompt Lab — analyze and optimize prompts using LLM-powered refiner/judge agents, accessible from trace viewer generation observations
  • Split-screen comparison — side-by-side view of original vs forked chat with shared messages dimmed and divergent messages highlighted

Details

Langfuse Observability

  • langfuse_helper.py manages client lifecycle (init/flush/shutdown) with lazy initialization
  • 10 agent lifecycle extensions cover the full trace: init, trace start, iteration, generation start/end, tool span start/end, utility calls, flush
  • Observability settings tab with connection test endpoint
  • Trace viewer modal with span/generation drill-down, token counts, cost, and timing

Chat Forking

  • fork_context() deep-copies via serialize/JSON-round-trip/deserialize with optional truncation at any log position
  • Fork buttons on message action bars and sidebar, fork indicator on forked chats
  • Auto-naming with collision check ("Chat (fork)", "Chat (fork 2)", etc.)

Prompt Lab

  • /prompt_refine generates 1-3 improved prompt variants via utility model
  • /prompt_judge scores variants on clarity, efficiency, fidelity, and safety (1-5 scale, rejection threshold < 3)
  • Three-panel modal: original prompt (read-only) | editor with suggest/test | results with scoring
  • Entry point from trace viewer on GENERATION observations

Split-Screen Comparison

  • /chat_logs read-only endpoint fetches logs from any context without switching active context
  • Fork point auto-detection from fork_info metadata with content-matching fallback
  • Shared messages rendered at 45% opacity, divergent messages highlighted with accent border
  • Compare button in sidebar for forked chats, also accessible from Prompt Lab

Files changed

40 files, +5,093 lines (26 new files, 14 modified)

image image image image image image image image

…-screen comparison

Langfuse Observability:
- Add langfuse_helper.py with client lifecycle management (init/flush/shutdown)
- Add 10 agent lifecycle extensions for tracing (init, trace, iteration, generation,
  tool spans, utility calls, flush)
- Add Observability settings tab with connection test endpoint
- Add per-interaction trace viewer modal with drill-down into spans/generations
- Migrate all extensions from Langfuse SDK v2 to v3 API

Chat Forking:
- Add fork_context() helper with serialization round-trip deep copy
- Add /chat_fork API endpoint with optional truncation at fork point
- Add forkChat() to frontend chats store with toast notifications
- Add fork buttons to message action buttons and sidebar chat list
- Add fork indicator icon for forked chats in sidebar

Prompt Lab:
- Add /prompt_refine endpoint with LLM-powered prompt variant generation
- Add /prompt_judge endpoint with multi-criteria scoring (clarity/efficiency/fidelity/safety)
- Add Prompt Lab modal with three-panel layout (original/editor/results)
- Add refine-then-judge pipeline with variant cards and score display
- Connect entry point from trace viewer's generation observations

Split-Screen Comparison:
- Add /chat_logs read-only endpoint for fetching logs from any context
- Add split-view store with parallel log fetching, fork point detection, and polling
- Add split-view component with shared (dimmed) vs divergent (highlighted) messages
- Add compare button to sidebar for forked chats
- Connect Prompt Lab's compareInSplitView() to split-screen overlay

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments