[scheduler/cuebot/pycue/rqd/pyoutline] Booking by slot limit#2101
Closed
DiegoTavares wants to merge 102 commits intoAcademySoftwareFoundation:new-schedulerfrom
Closed
[scheduler/cuebot/pycue/rqd/pyoutline] Booking by slot limit#2101DiegoTavares wants to merge 102 commits intoAcademySoftwareFoundation:new-schedulerfrom
DiegoTavares wants to merge 102 commits intoAcademySoftwareFoundation:new-schedulerfrom
Conversation
- Implement FrameRange and FrameSet structs to parse and represent complex frame range syntaxes including stepped, inverse stepped, negative steps, and interleaved ranges - Support chunking FrameSets into compact sub-ranges for dispatching - Integrate FrameSet chunking in RqdDispatcher for precise frame chunking - Improve dispatch error handling with distinct error types - Update host DAO and models to include allocation info for resource checks - Add .gitignore entry for /sandbox/kafka*
Signed-off-by: Diego Tavares <dtavares@imageworks.com>
The producer module produces events on kafka for each pending job. The consumer modules consume events and books jobs on host, still relying on the database.
This version still contains an issue when executing multiple tests at the same time, as tests are sharing a database instance an they rely on it existing to work.
Optimized async + pgpool interaction, but still far from perfect.
Last commit before giving up on dashmap
There is a protection against processing multiple bookings on a single host at the same time on HostDao that uses a database lock. This protection is intended for multiple instances of the scheduler running at the same time. However, this logic was also being triggered by a single instance, which indicated there was a race condition in place. The race condition happens because hosts can belong to multiple groups at the same time.
Use a central host store to prevent a split brain condition when a host belongs to multiple clusters at the same time.
Besides that, use host_stats for up-to-date memory information when updating the host cache.
To simplify testing, these changes are being migrated to a new PR
Entries were migrated to a new PR isolating the feature they were related to
The new option is define as: ```yaml ```
Signed-off-by: Diego Tavares <dtavares@imageworks.com>
This field limits the number of concurrent frames allowed to run on a specific host.
This commit is the first step towards the goal of allowing a new booking mode that doesn't take cores and memory into consideration, but a predefined limit on how many concurrent frames a host is allowed to run. Rationale: Booking by slot is useful for pipelines where frames are small and limited not by their cpu/memory consumption but by other resources like storage bandwith or network availability. In these scenarios, limiting the concurrency is more important than the resource consumption.
DiegoTavares
requested review from
lithorus and
ramonfigueiredo
as code owners
Add slots_required attribute to layer for slot-based booking
Collaborator
Author
|
A new PR has been created to handle the branch stacking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new booking mode that doesn't take cores and memory into consideration, but a predefined limit on how many concurrent frames a host is allowed to run.
Rationale: Booking by slot is useful for pipelines where frames are small and limited not by their
cpu/memory consumption but by other resources like storage bandwidth or network availability. In
these scenarios, limiting the concurrency is more important than the resource consumption.
Attention:* This branch is stacked on top of #2002
Tasks: