Swarming: Adds CountTasks endpoint & Refactors swarming Prpc request by IvanBM18 · Pull Request #5277 · google/clusterfuzz

IvanBM18 · 2026-05-14T23:05:56Z

As part of the Swarming backpressure initiative, we needed to launch request to the prpc's swarming.v2.Task/CountTasks endpoint so we can calculate the amount of fuzz tasks needed to schedule, to achieve this i moved the prpc request logic to its own module so that we can reuse code and keep responsibilities in separate files.

Changes

Created a new api.py file in the swarming module.
- Moved the request logic from __init__.py to api.py
- Removed push_swarming_task from init.py
- Updated service.py to use SwarmingAPI directly.
- Created a new class SwarmingAPI to handle the prpc request logic such as token/auth logic, swarming config handling, and any other request logic
Created new unit tests for this api and Updated existing unit tests.

Tests

Created unit tests for the new api & endpoint
Scheduled swarming jobs to verify that the refactor didn't broke the existing swarming logic
- See example logs

Note:

This PR is dependendant to this other PR any changes you see related to the proto files are to be reviewed in said pr and will merged trough that PR.

So please review this PR first then you can go on and review this one :D

Xeicker

For future PRs, when you move code and then add functionality it is a good practice to do each of those steps in separate commits for an easier review

… api

IvanBM18 · 2026-05-15T19:05:36Z

        ])
    self.assertEqual(spec, expected_spec)

-  def test_push_swarming_task(self):


Note: this test method is no longer needed here, in its place we added unit tests at the api_test that checks for the same functionality

This test exercised a much more complex NewTaskRequest, consider adding some of that back to api_test.py.

Xeicker

Other than some nits changes LGTM

letitz · 2026-05-19T12:11:06Z

  unique_dimensions = {}
  unique_dimensions['os'] = str(job.platform).capitalize()
-  unique_dimensions['pool'] = _get_swarming_config().get('swarming_pool')
+  unique_dimensions['pool'] = get_swarming_config().get('swarming_pool')


We already looked up the config on line 82. Let's use it:

Suggested change

unique_dimensions['pool'] = get_swarming_config().get('swarming_pool')

unique_dimensions['pool'] = swarming_config.get('swarming_pool')

letitz · 2026-05-19T12:14:46Z

+class SwarmingAPI:
+  """Client for Swarming pRPC API."""
+
+  _config: SwarmingConfig = None


Suggested change

_config: SwarmingConfig = None

_config: SwarmingConfig | None = None

letitz · 2026-05-19T12:17:13Z

+  def _get_api(self) -> SwarmingAPI:
+    """Returns the Swarming API instance."""
+    if not self._api:
+      self._api = SwarmingAPI()


Why instantiate this lazily? Let's just instantiate it at construction time, it's less surprising.

letitz · 2026-05-19T12:31:57Z

+      A dict containing headers, or empty dict if config is missing or
+      auth fails.
+    """
+    if not self._config:


If self._config is None, then this instance is useless. All its methods will just do nothing, and just log errors when invoked. It's a valid design choice, but it's a bit surprising: this is an API client, unless it's not and then nothing works!

I suggest giving this class a factory function that can fail if the config is missing, and exposing that failure to callers:

@staticmethod def create() -> 'SwarmingAPI' | None: config = get_swarming_config() if config is None: return None return SwarmingAPI(config) def __init__(self, config: SwarmingConfig): self._config = config self._base_url = ...

Then you can use that in SwarmingService to create an optional _api = SwarmingAPI | None member. Alternatively, you can inline the contents of create() in SwarmingService.__init__().

At that point, you've just moved the problem up one layer (now SwarmingService might be useless) but to my sense it's still cleaner.

Now, I think that to make this truly clean you'd want to go even one layer higher and only instantiate SwarmingService in RemoteTaskGate if swarming is enabled, and fail loudly if the config is missing (presumably if swarming is enabled, we can expect to have a SwarmingConfig).

I see that SwarmingService is unconditionally instantiated by RemoteTaskGate as part of its dict of adapters 1. I suggest only instantiating adapters if the relevant feature flag is enabled:

self._service_map = { adapter.id: adapter.service() for adapter in remote_task_adapters.RemoteTaskAdapters if adapter.feature_flag.enabled # new }

Then you only create instances of classes at runtime that you actually expect to use. This further limits the blast radius of changes to swarming code: if there is a bug in SwarmingService.__init__(), it won't be triggered in ClusterFuzz instances where Swarming is disables.

Finally, you can assert/crash if service instances cannot be constructed properly. In our case, you could assert in SwarmingService.__init__() that SwarmingAPI.create() succeeded, and make SwarmingService._api a non-optional member of type SwarmingAPI.

letitz · 2026-05-19T12:50:01Z

+      mock_config.side_effect = ValueError('Failed to load')
+      api = SwarmingAPI()
+      response = api.push_task(swarming_pb2.NewTaskRequest())
+      self.assertIsNone(response)


If you keep this design, you should check that no call was made to post_url().

letitz · 2026-05-19T12:50:23Z

+  def test_push_task_no_credentials(self):
+    """Tests that push_task fails when credentials are missing."""
+    self.mock.get_scoped_service_account_credentials.return_value = None
+    response = self.api.push_task(swarming_pb2.NewTaskRequest())


We should check that post_url() was not called.

letitz · 2026-05-19T12:50:30Z

+  def test_count_tasks_no_credentials(self):
+    """Tests that count_tasks fails when credentials are missing."""
+    self.mock.get_scoped_service_account_credentials.return_value = None
+    response = self.api.count_tasks(swarming_pb2.TasksCountRequest())


letitz · 2026-05-19T12:51:19Z

    helpers.patch(self, [
        'clusterfuzz._internal.swarming.is_swarming_task',
-        'clusterfuzz._internal.swarming.push_swarming_task',
+        'clusterfuzz._internal.swarming.service.SwarmingService._get_api',


If you define SwarmingApi.create(), you can mock that instead of a private helper.

letitz · 2026-05-19T12:52:22Z

        ])
    self.assertEqual(spec, expected_spec)

-  def test_push_swarming_task(self):


This test exercised a much more complex NewTaskRequest, consider adding some of that back to api_test.py.

IvanBM18 added 2 commits May 14, 2026 22:02

Add TasksCountRequest and TasksCount to swarming.proto

fc94459

Implement CountTasks and refactor Swarming API

56be2f4

IvanBM18 requested a review from a team as a code owner May 14, 2026 23:05

IvanBM18 changed the title ~~feature/swarming/count task requests~~ Swarming: Adds CountTasks endpoint & Refactors swarming Prpc request May 14, 2026

IvanBM18 requested review from Xeicker, fernandofloresg, jardondiego and letitz May 14, 2026 23:14

Xeicker reviewed May 15, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/swarming/api.py

Comment thread src/clusterfuzz/_internal/swarming/api.py Outdated

Comment thread src/clusterfuzz/_internal/swarming/api.py Outdated

Comment thread src/clusterfuzz/_internal/swarming/api.py

Removes push_swarming_task method & reuses get_swarming_config at the…

d78a8eb

… api

IvanBM18 commented May 15, 2026

View reviewed changes

Adds api as a SwarmingService property

0ede2ed

IvanBM18 self-assigned this May 15, 2026

IvanBM18 added the swarming Changes related to the clusterfuzz-swarming integration label May 15, 2026

IvanBM18 force-pushed the feature/swarming/count_task_requests branch from d45af03 to 562ec1b Compare May 15, 2026 20:20

Fixes Broken UT

57886fc

IvanBM18 force-pushed the feature/swarming/count_task_requests branch from 562ec1b to 57886fc Compare May 15, 2026 21:17

IvanBM18 requested a review from Xeicker May 15, 2026 21:41

Xeicker approved these changes May 15, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/swarming/api.py

Comment thread src/clusterfuzz/_internal/swarming/api.py Outdated

Adds endpoint constants

5a430c3

Xeicker approved these changes May 18, 2026

View reviewed changes

IvanBM18 added 2 commits May 18, 2026 18:53

Fixes linter issues

6b5715d

Merge branch 'master' into feature/swarming/count_task_requests

5299ee9

jardondiego reviewed May 18, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/protos/swarming.proto

IvanBM18 requested review from ViniciustCosta and javanlacerda May 18, 2026 20:14

IvanBM18 marked this pull request as draft May 18, 2026 22:10

letitz requested changes May 19, 2026

View reviewed changes

	unique_dimensions['pool'] = get_swarming_config().get('swarming_pool')
	unique_dimensions['pool'] = swarming_config.get('swarming_pool')

	_config: SwarmingConfig = None
	_config: SwarmingConfig \| None = None

Conversation

IvanBM18 commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Note:

Uh oh!

Xeicker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

IvanBM18 May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Xeicker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IvanBM18 commented May 14, 2026 •

edited

Loading

IvanBM18 May 15, 2026 •

edited

Loading