fix(storage): rate limit bucket deletion in cleanup#5612
Open
abhinavgautam01 wants to merge 1 commit intogoogleapis:mainfrom
Open
fix(storage): rate limit bucket deletion in cleanup#5612abhinavgautam01 wants to merge 1 commit intogoogleapis:mainfrom
abhinavgautam01 wants to merge 1 commit intogoogleapis:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request modifies the cleanup_stale_buckets function in the storage examples to serialize bucket deletions. The implementation replaces concurrent task spawning with a sequential loop that introduces a 2-second delay between deletions to comply with GCP rate limits. Feedback was provided to replace println! calls with the tracing crate to align with the repository's structured logging standards.
Serialize bucket deletion to respect GCP API rate limit (~1 request per 2 seconds). Uses structured logging with tracing crate for consistency with repository standards.
9c33688 to
1294c4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5219
Problem
When there are many stale buckets in integration tests, the cleanup process
was deleting them in parallel and exceeding GCP's Storage API rate limit
(approximately one request every two seconds).
Solution
Serialize bucket deletion by removing parallel spawning (tokio::spawn and
join_all) and instead delete buckets sequentially with a 2-second delay
between each deletion to respect the API rate limit.
Changes
cleanup_stale_buckets()to delete buckets sequentiallyTesting
This change should prevent rate limit errors during stale bucket cleanup
in integration tests.