Celery integration strips user custom headers from `task.request.headers`

[jamesrusso]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.53.0

Steps to Reproduce

Celery integration strips user custom headers from task.request.headers

Environment

Package	Version
sentry-sdk	2.53.0
celery	5.6.2
python	3.12

Description

The Celery integration's _update_celery_task_headers function in sentry_sdk/integrations/celery/__init__.py strips user-provided custom headers from reaching task.request.headers on the worker side.

Celery has a long-standing bug (celery/celery#4875) where task headers placed at the top level of the message are not reliably accessible via task.request.headers. Instead, only headers nested inside the inner "headers" dict (i.e., message["headers"]["headers"]) survive to task.request.headers on the worker.

The Sentry integration is aware of this issue -- there is even a comment referencing #4875 at line 219. The integration correctly places its own tracing headers (sentry-trace, baggage, etc.) into the inner "headers" dict. However, it does not copy user-provided custom headers into the inner dict, effectively making them invisible to task.request.headers on the worker.

Root Cause

In sentry_sdk/integrations/celery/__init__.py, the _update_celery_task_headers function (lines 162-234):

Step 1 (line 169): The function copies all original headers (including user custom headers) into updated_headers:

updated_headers = original_headers.copy()

Step 2 (line 224): It creates the inner "headers" dict and populates it with only Sentry's tracing headers (sentry-trace, baggage, etc.):

updated_headers.setdefault("headers", {}).update(headers)

Here, headers is the dict returned by sentry_sdk.get_isolation_scope().iter_trace_propagation_headers() -- it contains only Sentry tracing headers, not user custom headers.

Step 3 (lines 230-232): It then copies only keys prefixed with "sentry-" from the outer dict into the inner dict:

for key, value in updated_headers.items():
    if key.startswith("sentry-"):
        updated_headers["headers"][key] = value

This loop explicitly filters to "sentry-" prefixed keys. Any user-provided custom headers (e.g., my_custom_key, correlation_id, tenant_id) are left stranded at the top level and never copied into the inner "headers" dict.

Result: Due to celery/celery#4875, the worker's task.request.headers only reflects the inner dict contents. User custom headers are silently dropped.

Steps to Reproduce

Minimal reproduction script

# repro.py
import sentry_sdk
from sentry_sdk.integrations.celery import CeleryIntegration
from celery import Celery
from celery.signals import task_prerun

sentry_sdk.init(
    dsn="https://examplePublicKey@o0.ingest.sentry.io/0",
    integrations=[CeleryIntegration()],
    traces_sample_rate=1.0,
)

app = Celery("repro", broker="redis://localhost:6379/0")


@app.task(bind=True)
def my_task(self):
    # This is what users expect to work:
    value_from_headers = self.request.headers.get("my_custom_key")
    print(f"self.request.headers.get('my_custom_key') = {value_from_headers}")
    # ^ Returns None (BUG)

    # Workaround -- the value IS present at the top level of the request:
    value_from_attr = getattr(self.request, "my_custom_key", None)
    print(f"getattr(self.request, 'my_custom_key') = {value_from_attr}")
    # ^ Returns 'my_value'

    # Show what's actually in headers:
    print(f"self.request.headers = {self.request.headers}")
    # ^ Only contains sentry-trace, baggage, sentry-task-enqueued-time, etc.


if __name__ == "__main__":
    # Send the task with a custom header
    my_task.apply_async(args=[], headers={"my_custom_key": "my_value"})
    print("Task sent with headers={'my_custom_key': 'my_value'}")

Steps

1. Start a Redis instance (or any broker).
2. Run the worker: celery -A repro worker --loglevel=info
3. Send the task: python repro.py
4. Observe the worker output:

   self.request.headers.get('my_custom_key') = None
   getattr(self.request, 'my_custom_key') = my_value
   self.request.headers = {'sentry-trace': '...', 'baggage': '...', 'sentry-task-enqueued-time': ...}
   

5. Disable the Sentry integration (remove sentry_sdk.init(...)) and repeat.
6. Observe the worker output now correctly shows:

   self.request.headers.get('my_custom_key') = my_value
   

Expected Behavior

Custom headers passed via the headers= kwarg should be available in task.request.headers on the worker, regardless of whether the Sentry integration is enabled. The Sentry integration should not alter the visibility of user-provided headers.

Actual Behavior

When the Sentry integration is enabled, user custom headers are silently dropped from task.request.headers. They exist at the top level of the message (accessible via getattr(self.request, 'key')) but not in the inner "headers" dict that Celery exposes as task.request.headers.

Suggested Fix

In _update_celery_task_headers, after copying Sentry's own headers into the inner dict, also copy user-provided custom headers from original_headers so they survive the celery/celery#4875 behavior.

Option A: Copy all non-internal keys from original headers

After line 232, add:

# Preserve user-provided custom headers in the inner "headers" dict
# so they survive to task.request.headers on the worker (celery#4875).
for key, value in original_headers.items():
    if key != "headers" and key not in updated_headers.get("headers", {}):
        updated_headers["headers"][key] = value

Option B: Remove the "sentry-" prefix filter

Replace the loop at lines 230-232:

# Before (only copies sentry-* keys):
for key, value in updated_headers.items():
    if key.startswith("sentry-"):
        updated_headers["headers"][key] = value

# After (copies all non-internal keys):
for key, value in updated_headers.items():
    if key != "headers" and not isinstance(value, dict):
        updated_headers["headers"][key] = value

Option A is preferred because it uses original_headers as the source (avoiding accidentally copying Sentry's top-level additions like sentry-task-enqueued-time twice) and respects any keys the inner dict already has via setdefault.

Workaround

Until this is fixed, users can access custom headers via top-level request attributes instead of task.request.headers:

@app.task(bind=True)
def my_task(self):
    # Instead of:
    # value = self.request.headers.get("my_custom_key")

    # Use:
    value = getattr(self.request, "my_custom_key", None)

Note: this workaround is fragile and relies on Celery's internal behavior of flattening top-level message headers into request attributes.

References

• celery/celery#4875 -- the upstream Celery bug that necessitates the inner "headers" dict workaround
• sentry_sdk/integrations/celery/__init__.py lines 162-234 -- the _update_celery_task_headers function

Expected Result

The headers included on the request are preserved.

Actual Result

The headers included on the request are stripped out.

[linear]

PY-2118

[ericapisani]

Hi @jamesrusso ,

I've pulled this into the team's backlog, thanks for reaching out and submitting such a detailed description/reproduction.