Context
Discovered while fixing #146 (Cohere) and #147 (Llama-3) on the Python SDK side. The SDK's event-stream parser now correctly decodes AWS Bedrock framing and extracts text via the configured stream_response_path JSONPath. But for streaming to actually yield text, the route's ModelSpec.stream_response_path must match the chunk schema emitted by that model.
Each Bedrock model family emits a different chunk shape, so a single default (currently delta.text) silently breaks the rest.
Known per-model chunk schemas
| Model family |
Correct stream_response_path |
| Anthropic Claude |
delta.text |
| Meta Llama (3.x, 2) |
generation |
| Cohere Command |
text |
| Mistral (Bedrock) |
outputs[0].text |
| AI21 Jamba |
choices[0].delta.content |
| Amazon Titan Text |
outputTokens[0].text (verify) |
Ask
- Audit existing routes/ModelSpecs for Bedrock-hosted models and confirm each carries the correct
stream_response_path.
- Set sensible per-family defaults in whatever creates ModelSpecs so new routes get the right path automatically.
- Add a startup/validation check that flags routes pointing at a known Bedrock model family with an obviously wrong
stream_response_path.
Why this is a separate ticket
The SDK change (event-stream parser rewrite, PR linked to #146/#147) is necessary but not sufficient. Even with a correct parser, a misconfigured stream_response_path produces an empty stream — now with a structured WARNING log naming the path that missed and the payload keys observed, so this audit is actionable.
Context
Discovered while fixing #146 (Cohere) and #147 (Llama-3) on the Python SDK side. The SDK's event-stream parser now correctly decodes AWS Bedrock framing and extracts text via the configured
stream_response_pathJSONPath. But for streaming to actually yield text, the route'sModelSpec.stream_response_pathmust match the chunk schema emitted by that model.Each Bedrock model family emits a different chunk shape, so a single default (currently
delta.text) silently breaks the rest.Known per-model chunk schemas
stream_response_pathdelta.textgenerationtextoutputs[0].textchoices[0].delta.contentoutputTokens[0].text(verify)Ask
stream_response_path.stream_response_path.Why this is a separate ticket
The SDK change (event-stream parser rewrite, PR linked to #146/#147) is necessary but not sufficient. Even with a correct parser, a misconfigured
stream_response_pathproduces an empty stream — now with a structured WARNING log naming the path that missed and the payload keys observed, so this audit is actionable.