Skip to content

[Swarming] Add backpressure mechanism to SwarmingService#5284

Draft
IvanBM18 wants to merge 9 commits into
masterfrom
feature/swarming/backpressure
Draft

[Swarming] Add backpressure mechanism to SwarmingService#5284
IvanBM18 wants to merge 9 commits into
masterfrom
feature/swarming/backpressure

Conversation

@IvanBM18
Copy link
Copy Markdown
Collaborator

@IvanBM18 IvanBM18 commented May 19, 2026

Overview

This PR implements a backpressure mechanism in SwarmingService to prevent overloading the Swarming pool with too many tasks.

Changes

  • Added MAX_PENDING_TASKS = 50 to limit the number of pending tasks.
    • This threshold was chosen considering a pool of at most 100 bots and a scheduler running every 2.5 minutes. It ensures bots don't go idle (having at least ~5 tasks in the queue at all time)
  • Updated create_utask_main_jobs to check the queue size before pushing each task using Swarming's CountTasks RPC.
  • Implemented a Fail Closed strategy: if the CountTasks rpc call fails, we assume the queue is full and stop scheduling further tasks in the batch.
  • Added unit tests to verify backpressure and failure handling.

Note

This PR is dependant on this other PR so please go ahead and review that one first :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant