chore(source-google-ads): pin CDK to unlimited queue prerelease for deadlock workaround#74763
chore(source-google-ads): pin CDK to unlimited queue prerelease for deadlock workaround#74763devin-ai-integration[bot] wants to merge 3 commits intomasterfrom
Conversation
…ue size 50K) Co-Authored-By: unknown <>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
|
|
…imited queue) Co-Authored-By: unknown <>
devin-ai-integration
Bot
changed the title
|
…imited queue fix) Updates CDK prerelease to include fix for broken ThreadPoolManager futures limit check that was caused by DEFAULT_MAX_QUEUE_SIZE=0. Co-Authored-By: unknown <>
|
1 participant
What
Pins
source-google-adsto a CDK prerelease (7.12.0.post3.dev23019060129) that makes the internal record queue unlimited (maxsize=0). This is a temporary workaround for a deadlock observed in Airbyte Cloud where the platform pauses reading from the source container's stdout pipe, causing all worker threads to block onqueue.put().Related CDK PR: airbytehq/airbyte-python-cdk#951
How
airbyte-cdkpin inpyproject.tomlfrom7.9.2→7.12.0.post3.dev23019060129poetry.lockNo connector code changes. No version bump (intentional — this is for prerelease testing only).
Updates since last revision
7.12.0.post2.dev23017457367→7.12.0.post3.dev23019060129. The previous unlimited queue prerelease had a critical bug: settingDEFAULT_MAX_QUEUE_SIZE = 0also brokeThreadPoolManagerbecause that constant is used as the default formax_concurrent_tasks. Withmax_concurrent_tasks=0, the futures limit check (len(futures) >= max_concurrent_tasks) was always true, blocking all worker thread submissions — resulting in zero records emitted. The new version restoresDEFAULT_MAX_QUEUE_SIZE = 10_000(for the futures limit) while keepingQueue(maxsize=0)(unlimited queue size).Review guide
pyproject.toml— the only substantive change (CDK version pin)poetry.lock— auto-generated from the version pinKey concern: This jumps from stable
7.9.2to a7.12.0dev prerelease. Reviewer should verify whether any breaking changes between 7.9.2 and 7.12.0 affect this connector. See CDK migration guide.Review & Testing Checklist for Human
maxsize=0) means potentially unbounded memory growth if workers produce records faster than the main thread consumes them. If the platform stops reading stdout indefinitely, memory could grow until the container OOMs. Confirm this risk is acceptable vs. the deadlock risk for this prerelease test.source-google-ads.maxsize=0) frommax_concurrent_tasksdefault (10_000).User Impact
No user impact — this PR is intended solely for publishing a connector prerelease to test the unlimited queue workaround on a sandbox connection. It should not be merged as-is for a production release.
Can this PR be safely reverted and rolled back?
Requested by @AnatoliiYatsuk.
Link to Devin run