Retry 413 / payload too large errors in OTLP batch exporter

[adriangb]

Is your feature request related to a problem?

Many backends have size limits. If you try to send too large of a payload it may get dropped on the floor. This is somewhat mitigated by setting a batch size limit (in terms of number of spans) but that (1) is not at the byte size level and (2) you need to know ahead of time what your backend's limit is.

Describe the solution you'd like

I would the SDK to retry on this failure code by reducing the payload size (e.g. split it into two exports) until it's accepted or a single span is rejected.

Ideally we'd have a way for the backend to communicate back it's limit and the SDK could even store that and avoid hitting the error in the future, but I think that would require agreement at the OTEL spec level for an expected response payload body that includes the maximum allowed size. Implementing a retry in the Python SDK can be a good step in that direction.

Describe alternatives you've considered

No response

Additional Context

No response

Would you like to implement a fix?

Yes

[tammy-baylis-swi]

Hi @adriangb , please could you describe a bit more about your use case(s)?

There is some Batch Splitting (max_export_batch_size) implemented now for the OTel Python OTLP exporters, if you haven't already tried it. There is some retry logic as well.

I will note that both client-side strategies are inconsistent in the current OTel Python implementations, and under the hood are a little intertwined. iiuc:

Exporter Type	413/RESOURCE_EXHAUSTED Retryable?	Batch Splitting (max_export_batch_size)
gRPC Metrics	✅ Yes (RESOURCE_EXHAUSTED retryable)	✅ Yes
gRPC Traces	✅ Yes (RESOURCE_EXHAUSTED retryable)	❌ No
gRPC Logs	✅ Yes (RESOURCE_EXHAUSTED retryable)	❌ No
HTTP Metrics	❌ No (retryable, but not for 413)	✅ Yes
HTTP Traces	❌ No (retryable, but not for 413)	❌ No
HTTP Logs	❌ No (retryable, but not for 413)	❌ No

Given PR comment (thanks!), I was hesitant about some proposed changes initially and because the spec is Stable. But some well-scoped changes might help given what some exporter features already exist. Would be good to get others' opinions too.