feat(telemetry): normalise cache and reasoning token fields in GenerationMetadata across providers

[planetf1]

Problem

record_token_usage in mellea/telemetry/backend_instrumentation.py extracts cache and reasoning token counts using get_value, which does not walk nested paths. As a result:

• Anthropic: cache_read_input_tokens and cache_creation_input_tokens are top-level in the usage object — extracted correctly today.
• OpenAI / LiteLLM: cache tokens live at prompt_tokens_details.cached_tokens and reasoning tokens at completion_tokens_details.reasoning_tokens — silently absent from spans because get_value cannot traverse the nesting.

This means gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens are never emitted for OpenAI or LiteLLM backends, with no warning to the user.

A secondary concern: the attribute names themselves (gen_ai.usage.cache_read.input_tokens etc.) are Anthropic-specific and not yet defined in the upstream OTel GenAI semconv — see #1054 for the naming audit.

Why not fix in the current extraction layer

Patching get_value for dotted-path walking or adding provider-specific branches in record_token_usage is short-lived work: after #1045, record_token_usage will read from mot.generation (GenerationMetadata) rather than raw provider response objects, making any provider-specific path logic obsolete.

Approach

During or after #1045, ensure GenerationMetadata (or the per-backend post-processing that populates it) normalises cache and reasoning token fields to a flat, provider-agnostic shape. record_token_usage then reads the normalised struct without any provider-specific path walking.

Specifically:

• OpenAI backend post_processing(): flatten prompt_tokens_details.cached_tokens → mot.usage["cache_read_input_tokens"]
• OpenAI backend post_processing(): flatten completion_tokens_details.reasoning_tokens → mot.usage["reasoning_tokens"]
• Verify LiteLLM normalisation path does the same (LiteLLM may already flatten this)
• Confirm attribute names align with OTel GenAI semconv at the time of implementation (coordinate with test: backend coverage parity audit post-refactor #1054)

Prerequisites

• refactor: move tracing onto plugin/hook pattern #1045 (plugin/hook refactor) — GenerationMetadata normalisation is Phase 1 work

Related

• Epic Epic: Enhanced distributed tracing #444 Phase 1 / refactor: move tracing onto plugin/hook pattern #1045
• test: backend coverage parity audit post-refactor #1054 (attribute naming audit)
• PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 (where the gap was surfaced in review — Anthropic path works, OpenAI path silently drops fields)