Bulter.py: Adds preprocess butler script for local preprocess

[IvanBM18]

Adds preprocess butler script

This butler script allows developers to trigger the preprocess portion of a fuzz task and in consecuence generate the serialized and compressed uworker_input payload, upload it to real GCS, and get the signed download URL, exactly as it happens remotely. We can then use the resulting url to trigger a task in any backend that we want:

• In swarming trough a prpc request
• In batch trough manually posting the task to the utask_main queue

This accelerates local debugging of the tworker preprocessing phase without relying on remote execution queues, which has proven to take multiple hours to "ACK" a task request.

Note: To use this command you need the Secret Manager Secret Accessor for Dev or setup a service account key in your local(by using the gcloud auth cli) that has said role and any other role required for a tworker's preprocess.

Example:

With an input like this

pipenv run python butler.py run preprocess --config-dir ~/clusterfuzz-config/configs/ --script_args fuzzer linux_asan_fuzzer

You should get an output like this:

.......
Preprocess successful!
Input Download URL: https://storage.googleapis.com/uworker-input.project/.....

Changes

• Added the preprocess script.
  • It interacts with the actual Datastore and GCS based on the provided configuration.
  • It fetches and populates the uworker_env with:
    • Job-specific environment variables from the Datastore.
    • Fuzzer-specific environment variables (for blackbox fuzzers).
    • Required logging metadata (CF_TASK_NAME, CF_TASK_ARGUMENT, CF_TASK_JOB_NAME, CF_TASK_ID) to ensure logs in the subsequent uworker_main step have the correct context as if we executed it using the usual commands.py module workflow.

Tests performed

Executed the following command in dev:

pipenv run python butler.py run preprocess -c <config_dir>  --script_args <fuzzer> <job>

Successfully creates and uploads the payload and returns a valid signed URL. This signed url was later used to trigger a swarming task trough prpc, here are the logs

[jardondiego]

Note: To use this command you need the Secret Manager Secret Accessor for Dev or setup a service account in your local that has those permissions.

Did you use a .json service account for this? It would be nice to have a few sentences/links to understand it better.

[jardondiego]

Does it need to be a subcommand of butler itself? Why not make it a standalone script? I think it's a good idea to not over-populate butler with subcommands. What do you think?

[letitz]

Could we use process_command_impl() then instead?

[IvanBM18]

At the end of said method we call run_command():
https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/tasks/commands.py#L482
Which in turns triggers a workflow in which the preprocess step immediately queues the main task for remote execution when finished or just straight ups executes all 3 steps in the same machine(depending on setup), but we don't want that, we want to stop just after finishing the preprocess so we could manually trigger the main portion wherever and whenever we need to

[letitz]

I see, thanks. Then this works for me. It will inevitably drift apart from the prod implementation, especially since this codepath is not exercised in production, but it's good enough to run for now and commit for others to reuse :)

[IvanBM18]

@jardondiego

Note: To use this command you need the Secret Manager Secret Accessor for Dev or setup a service account in your local that has those permissions.

Did you use a .json service account for this? It would be nice to have a few sentences/links to understand it better.

Not in this case, but its possible to use a service account, you just need to generate a key, save it in your local and set it up as the default credentials for any gcloud library and cli operation. This is done using the gcloud auth subcommand.

Added more context in the description so future reviewers can easily understand this

[IvanBM18]

@jardondiego

Does it need to be a subcommand of butler itself? Why not make it a standalone script? I think it's a good idea to not over-populate butler with subcommands. What do you think?

Yes, its need to as butler already handles a lot of bootstrapping operations for the same purpose, for example if we didn't use butler, we would need to add methods to read, parse, and populate configurations based on the yaml files

[jardondiego]

Yes, its need to as butler already handles a lot of bootstrapping operations for the same purpose, for example if we didn't use butler, we would need to add methods to read, parse, and populate configurations based on the yaml files

I think we are referring to different things, what I mean is that I think we could have as a standalone butler script so that we can run it with as

python butler.py run <name_of_script> --non-dry-run --config $MY_DIR

That way we don't have to handle all of that by ourselves. Does it make sense?

[jardondiego]

Added more context in the description so future reviewers can easily understand this

Thanks!

[jardondiego]

LGTM

[letitz]

LGTM as a testing tool, not production code.