Job throughput with concurrency limits

[dm5991]

Hi, we recently moved to River Pro version and decided to add global limits for one of our queues. The queue is processed by 4 client instances that each have 10 max workers and the only change in the configuration is the added global limit of 40.

The jobs themselves are pretty fast at an average of ~5 seconds. Before adding the global limits we used to see a throughput of ~33k jobs per hour and afterwards it dropped to ~17k per hour. A couple of immediate tests with a larger worker pool didn't make any difference.

We would expect a slowdown to a certain degree from the added coordination overhead combined with fast jobs but this seems a bit extreme.
Do you have any info on what the expected throughput slowdowns are when using concurrency limits? Happy to give more info if it helps, thanks!

River versions:

	github.com/riverqueue/river v0.30.2
	github.com/riverqueue/river/riverdriver/riverpgxv5 v0.30.2

	riverqueue.com/riverpro v0.21.0
	riverqueue.com/riverpro/driver/riverpropgxv5 v0.21.0

[bgentry]

Hi @dm5991! In your setup where you have exactly the number of worker slots that you've configured as your global limit (4*10 = 40) and jobs that run for more than a second, I wouldn't expect to see much of an impact to throughput. Can you confirm whether you're using the default FetchCooldown of 100ms and FetchPollInterval of 1s?

The global concurrency limit feature uses a locking mechanism which means that only one client can fetch on that queue at a given moment (in order to absolutely guarantee the limits are not exceeded at any moment) but the overhead on this is typically very small, particularly with only a handful of clients. While I would expect a small hit to throughput due to internal buffering (primarily for completion) and added locking overhead, I wouldn't expect anything close to the ~50% drop you're seeing.

We have extensive benchmarking on the main queries involved here but I don't think we have any that test throughput across multiple clients in this kind of scenario. I'll put one together and report back whether I see anything similar or if we need to further narrow down what might be causing it on your side.

Also, just FYI as a Pro customer you can always email us directly at team@ in case you want to share more details outside a public issue tracker.

[bgentry]

Hi @dm5991, I did a bit more investigation and testing here. I wrote a benchmark that uses your exact settings (5s jobs, 4 clients, 10 max workers, default fetch cooldown 100ms, fetch poll interval 1s). The theoretical max for this is 28,800 jobs per hour but that's if the fetches themselves take 0 time.

With this setup, I'm seeing 26k-27k jobs/hr without concurrency limits enabled, and 22k-23k jobs/hr with a global limit of 40. That's definitely a drop, but not as severe as what you're seeing.

For context, when concurrency limits are enabled, each client is separately tracking its own in-progress jobs in the database. These counts get updated at the time that client attempts to fetch jobs on the queue in question. In order to guarantee these global limits fetches on a queue with a global concurrency limit must be done serially among the clients on that queue. This does add some overhead but I wouldn't expect it to be anywhere near 50%.

[dm5991]

Thanks for looking into this so fast!

I can confirm we are not setting anything for FetchCooldown and FetchPollInterval in either the client config or the queue specific config, so that should mean the defaults get used.

I've also tried running some local tests with no-op jobs for this queue to see if I can debug something and I'm seeing very similar throughput slowdown as you did (~15%), so I'm going to dig a bit deeper into some of the server and DB configs.
I'll update the issue if I find anything but in the meantime thank you for confirming this amount of slowdown shouldn't be happening.