Skip to content

Bedrock: audit ModelSpec.stream_response_path defaults per model family #251

@safayavatsal

Description

@safayavatsal

Context

Discovered while fixing #146 (Cohere) and #147 (Llama-3) on the Python SDK side. The SDK's event-stream parser now correctly decodes AWS Bedrock framing and extracts text via the configured stream_response_path JSONPath. But for streaming to actually yield text, the route's ModelSpec.stream_response_path must match the chunk schema emitted by that model.

Each Bedrock model family emits a different chunk shape, so a single default (currently delta.text) silently breaks the rest.

Known per-model chunk schemas

Model family Correct stream_response_path
Anthropic Claude delta.text
Meta Llama (3.x, 2) generation
Cohere Command text
Mistral (Bedrock) outputs[0].text
AI21 Jamba choices[0].delta.content
Amazon Titan Text outputTokens[0].text (verify)

Ask

  1. Audit existing routes/ModelSpecs for Bedrock-hosted models and confirm each carries the correct stream_response_path.
  2. Set sensible per-family defaults in whatever creates ModelSpecs so new routes get the right path automatically.
  3. Add a startup/validation check that flags routes pointing at a known Bedrock model family with an obviously wrong stream_response_path.

Why this is a separate ticket

The SDK change (event-stream parser rewrite, PR linked to #146/#147) is necessary but not sufficient. Even with a correct parser, a misconfigured stream_response_path produces an empty stream — now with a structured WARNING log naming the path that missed and the payload keys observed, so this audit is actionable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions