ci: Use GH concurrency queue instead of turnstile in build.yml#49068
Open
anomiex wants to merge 1 commit into
Open
ci: Use GH concurrency queue instead of turnstile in build.yml#49068anomiex wants to merge 1 commit into
anomiex wants to merge 1 commit into
Conversation
GitHub has recently added the ability to have an actual queue (max 100 entries) for its concurrency feature. This should perform better than turnstile, so let's try it out. To make this work, we have to stop using GitHub's matrixing for the `build` step, because the new `update_wpcom_mirrors` step needs to depend on only the wpcom build and not the other parts of the matrix. So instead we have to repeat the boilerplate for each of the three parts of the matrix, and use yaml anchors to avoid repeating all the steps too. This does dd some risk of out-of-order pushes. Say we have two trunk pushes, A and B, pushed in that order. * With turnstile, if B gets to `update_mirrors` while A is still on `build`, it'll wait on A. The only opportunity for B to run ahead of A is if B's once-per-minute check happens to run in between when A's `build` or `merge_artifacts` finishes and its next job (`merge_artifacts` or `update_mirrors`) gets queued. * With concurrency groups, it'll run them in the order that each `update_mirrors` is queued, with no regard for which workflow started first. The effect of out-of-order pushes is that B's run would push a squashed A+B commit, then A's run will find nothing to do. OTOH, on the plus side, this improves the deploy-to-Simple pipeline when there are a bunch of merges. Right now, B can't push the two plugins until A completely finishes building and pushing everything. With the concurrency queues, we can put the two push jobs into separate queues, so B can go ahead and start pushing Jetpack even if A is still working on all the non-wpcom stuff.
Contributor
|
Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.
Interested in more tips and information?
|
github-actions
Bot
added
the
Actions
Contributor
|
Thank you for your PR! When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:
This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖 Follow this PR Review Process:
If you have questions about anything, reach out in #jetpack-developers for guidance! |
Contributor
Author
|
I'm not sure I'm entirely happy with this one. It winds up with a bunch of extra skipped jobs in the PR output, that might be confusing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
GitHub has recently added the ability to have an actual queue (max 100 entries) for its concurrency feature. This should perform better than turnstile, so let's try it out.
To make this work, we have to stop using GitHub's matrixing for the
buildstep, because the newupdate_wpcom_mirrorsstep needs to depend on only the wpcom build and not the other parts of the matrix. So instead we have to repeat the boilerplate for each of the three parts of the matrix, and use yaml anchors to avoid repeating all the steps too.This does dd some risk of out-of-order pushes. Say we have two trunk pushes, A and B, pushed in that order.
update_mirrorswhile A is still onbuild, it'll wait on A. The only opportunity for B to run ahead of A is if B's once-per-minute check happens to run in between when A'sbuildormerge_artifactsfinishes and its next job (merge_artifactsorupdate_mirrors) gets queued.update_mirrorsis queued, with no regard for which workflow started first.The effect of out-of-order pushes is that B's run would push a squashed A+B commit, then A's run will find nothing to do.
OTOH, on the plus side, this improves the deploy-to-Simple pipeline when there are a bunch of merges. Right now, B can't push the two plugins until A completely finishes building and pushing everything. With the concurrency queues, we can put the two push jobs into separate queues, so B can go ahead and start pushing Jetpack even if A is still working on all the non-wpcom stuff.
Related product discussion/links
p1779371671037339-slack-C05Q5HSS013
Does this pull request change what data or activity we track or use?
No
Testing instructions