fix(python): remove hardcoded OTEL_LOG_LEVEL=DEBUG that caused stderr deadlocks#3377
fix(python): remove hardcoded OTEL_LOG_LEVEL=DEBUG that caused stderr deadlocks#3377SAY-5 wants to merge 1 commit intotriggerdotdev:mainfrom
Conversation
… deadlocks Per triggerdotdev#3357, `python.runScript()` deadlocks any Python subprocess that writes more than ~64KB to stderr. The previous implementation hardcoded `OTEL_LOG_LEVEL: 'DEBUG'` in the subprocess environment, which caused OTEL-aware Python libraries (mlflow, opentelemetry-sdk, etc.) to emit verbose debug-level logs to stderr during import. Once stderr exceeded the OS pipe buffer, the Python process would block on its next `write()` syscall indefinitely — a silent, permanent hang with no error or timeout (unless `maxDuration` was set). Remove the hardcoded `OTEL_LOG_LEVEL: 'DEBUG'`. This brings `runScript()` in line with `run()`, `runInline()`, and the streaming variants — none of which set `OTEL_LOG_LEVEL`. The original line was almost certainly leftover dev instrumentation that shipped accidentally. Closes triggerdotdev#3357
🦋 Changeset detectedLatest commit: de5c162 The changes in this PR will be included in the next version bump. This PR includes changesets to release 29 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughThis PR adds a changeset documenting a deadlock fix in Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Hi @SAY-5, thanks for your interest in contributing! This project requires that pull request authors are vouched, and you are not in the list of vouched users. This PR will be closed automatically. See https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md for more details. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
1 participant
Per #3357,
python.runScript()deadlocks any Python subprocess that writes more than ~64KB to stderr.The previous implementation hardcoded
OTEL_LOG_LEVEL: 'DEBUG'in the subprocess environment (line 100), which caused OTEL-aware Python libraries (mlflow, opentelemetry-sdk, etc.) to emit verbose debug-level logging to stderr during import. Once stderr exceeded the OS pipe buffer, the Python process blocked on its nextwrite()syscall indefinitely — a silent, permanent hang with no error or timeout (unlessmaxDurationwas set).This PR removes that single line, bringing
runScript()in line withrun(),runInline(), and the streaming variants — none of which setOTEL_LOG_LEVEL. The original line was almost certainly leftover dev instrumentation that shipped accidentally.A patch changeset is included.
Closes #3357