Skip to content

Add tool call tracing and logging for agent routing debugging #142

@neuromechanist

Description

@neuromechanist

Problem

When the EEGLAB assistant gives a suboptimal answer (e.g., missing the standalone erpimage function), developers have no way to trace what happened during the request:

  • Which tools did the model call?
  • What arguments were passed?
  • What did each tool return?
  • Did a tool return a fallback/error message?
  • Why did the model choose one tool over another?

Currently, tool calls are tracked in BaseAgentState.tool_calls and returned in ChatResponse.tool_calls, but there is no server-side logging of the tool execution flow.

Current State

In src/agents/base.py:99-118, _agent_node records tool calls into state but does not log:

def _agent_node(self, state: BaseAgentState) -> dict[str, Any]:
    messages = self._prepare_messages(state)
    response = self.model_with_tools.invoke(messages)
    # Tool calls silently tracked, no logging
    tool_calls = state.get("tool_calls", [])
    if hasattr(response, "tool_calls") and response.tool_calls:
        for tc in response.tool_calls:
            tool_calls.append({"name": tc["name"], "args": tc["args"]})
    return {"messages": [response], "tool_calls": tool_calls}

Proposed Changes

1. Log tool calls in _agent_node

Add logger.info when the model makes tool calls:

INFO agent: Tool call: search_eeglab_docstrings(query="erpimage", limit=5)
INFO agent: Tool call: retrieve_eeglab_docs(title="Plotting ERP images")

2. Log tool results in ToolNode or post-processing

Log success/failure and result length:

INFO agent: Tool result: search_eeglab_docstrings -> 5 results (1234 chars)
INFO agent: Tool result: search_eeglab_docstrings -> "Knowledge base not initialized..."

3. Log fallback/error tool responses

When tools return known error patterns (e.g., "Knowledge base not initialized", "No results found"), log at WARNING level so these stand out in logs.

4. Optional: Debug mode for API responses

Add a ?debug=true query parameter that includes the full tool execution trace in the API response, useful for development and troubleshooting without checking server logs.

Files Involved

  • src/agents/base.py - _agent_node, _should_use_tools
  • src/api/routers/community.py - API response construction
  • LangFuse integration (src/core/services/llm.py) is already implemented but optional; this issue is about basic logging that works without LangFuse.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Priority 2: Important, fix when possibledeveloper-experienceImproves developer experienceenhancementNew feature or requestobservabilityLogging, monitoring, and debugging

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions