From 9fd6c9a9b2b3b5421c797ea9a2ef0cc07442083f Mon Sep 17 00:00:00 2001 From: Lorenzo Buonanno Date: Fri, 16 Jan 2026 16:44:24 +0100 Subject: [PATCH 1/4] docs: document array payload merging behavior with job_key When both the existing and new job payloads are JSON arrays, they are concatenated rather than replaced. This undocumented feature enables powerful batching patterns, especially when combined with `preserve_run_at` for fixed batching windows. - Add "Array payload merging" section to job-key.md with example - Add cross-reference tip in tasks.md batch jobs section - Document the caution that non-array payloads trigger replace behavior Co-Authored-By: Claude Opus 4.5 --- website/docs/job-key.md | 39 +++++++++++++++++++++++++++++++++++++++ website/docs/tasks.md | 8 ++++++++ 2 files changed, 47 insertions(+) diff --git a/website/docs/job-key.md b/website/docs/job-key.md index 35ab76fa..1362f932 100644 --- a/website/docs/job-key.md +++ b/website/docs/job-key.md @@ -71,6 +71,45 @@ The full `job_key_mode` algorithm is roughly as follows: - Otherwise: - the job will have all its attributes updated to their new values. +### Array payload merging + +When updating an existing job via `job_key`, if both the existing job's +payload and the new payload are JSON arrays, they will be **concatenated** +rather than replaced. This enables a batching pattern where multiple +events can be accumulated into a single job. + +```sql +-- First call creates job with payload: [{"id": 1}] +SELECT graphile_worker.add_job( + 'process_events', + '[{"id": 1}]'::json, + job_key := 'my_batch', + job_key_mode := 'preserve_run_at', + run_at := NOW() + INTERVAL '10 seconds' +); + +-- Second call (before job runs) merges to: [{"id": 1}, {"id": 2}] +SELECT graphile_worker.add_job( + 'process_events', + '[{"id": 2}]'::json, + job_key := 'my_batch', + job_key_mode := 'preserve_run_at', + run_at := NOW() + INTERVAL '10 seconds' +); +``` + +Combined with `preserve_run_at` job_key_mode, this creates a fixed batching window: the job +runs at the originally scheduled time with all accumulated payloads merged +together. With the default `replace` job_key_mode, each new event would push the +`run_at` forward, creating a rolling/debounce window instead. + +:::caution + +If **either** payload is not an array (e.g., one is an object), the standard +replace behavior applies and the old payload will be lost. + +::: + ## Removing jobs Pending jobs may also be removed using `job_key`: diff --git a/website/docs/tasks.md b/website/docs/tasks.md index a419ae7d..7c350d78 100644 --- a/website/docs/tasks.md +++ b/website/docs/tasks.md @@ -193,6 +193,14 @@ any of these promises reject, then the job will be re-enqueued, but the payload will be overwritten to only contain the entries associated with the rejected promises — i.e. the successful entries will be removed. +:::tip Accumulating batch payloads with job_key + +You can use [`job_key`](./job-key.md#array-payload-merging) with array payloads +to accumulate multiple events into a single batch job. When both the existing +and new payloads are arrays, they are concatenated automatically. + +::: + ## `helpers` ### `helpers.abortPromise` From 204a18e3c45860656ab9a26c31e420cb2bb4c7f3 Mon Sep 17 00:00:00 2001 From: Lorenzo Buonanno Date: Wed, 25 Feb 2026 00:26:20 +0100 Subject: [PATCH 2/4] docs: clarify job_key payload behavior unsafe_dedupe mode and no payload --- website/docs/job-key.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/website/docs/job-key.md b/website/docs/job-key.md index 1362f932..ad8ec6c2 100644 --- a/website/docs/job-key.md +++ b/website/docs/job-key.md @@ -76,7 +76,7 @@ The full `job_key_mode` algorithm is roughly as follows: When updating an existing job via `job_key`, if both the existing job's payload and the new payload are JSON arrays, they will be **concatenated** rather than replaced. This enables a batching pattern where multiple -events can be accumulated into a single job. +events can be accumulated into a single job (except in `unsafe_dedupe` mode). ```sql -- First call creates job with payload: [{"id": 1}] @@ -106,7 +106,9 @@ together. With the default `replace` job_key_mode, each new event would push the :::caution If **either** payload is not an array (e.g., one is an object), the standard -replace behavior applies and the old payload will be lost. +replace behavior applies and the old payload will be lost. Note that the +default payload is an object, so calling `add_job(s)` without an explicit +payload will also trigger a replace rather than a concatenation. ::: From 0ea0b7c35ae61acfa6ca80873b76da03441d6ee8 Mon Sep 17 00:00:00 2001 From: Benjie Gillam Date: Fri, 27 Feb 2026 10:44:50 +0000 Subject: [PATCH 3/4] Edits to documentation --- website/docs/job-key.md | 62 +++++++++++++++++++++++++++-------------- website/docs/tasks.md | 5 ++-- 2 files changed, 44 insertions(+), 23 deletions(-) diff --git a/website/docs/job-key.md b/website/docs/job-key.md index ad8ec6c2..1693fb36 100644 --- a/website/docs/job-key.md +++ b/website/docs/job-key.md @@ -29,6 +29,17 @@ SELECT graphile_worker.add_job('send_email', '{"count": 2}', job_key := 'abc'); COMMIT; ``` +If both the previous and new versions of the job use array payloads they will be +merged; for example the `process_invoices` job will run once with the payload +`[{id: 42},{id: 67}]`: + +```sql +BEGIN; +SELECT graphile_worker.add_job('process_invoices', '[{"id": 42}]', job_key := 'inv'); +SELECT graphile_worker.add_job('process_invoices', '[{"id": 67}]', job_key := 'inv'); +COMMIT; +``` + In all cases if no match is found then a new job will be created. ### `job_key_mode` @@ -36,14 +47,15 @@ In all cases if no match is found then a new job will be created. Behavior when an existing job with the same job key is found is controlled by the `job_key_mode` setting: -- `replace` (default) - overwrites the unlocked job with the new values. This is - primarily useful for rescheduling, updating, or **debouncing** (delaying - execution until there have been no events for at least a certain time period). - Locked jobs will cause a new job to be scheduled instead. -- `preserve_run_at` - overwrites the unlocked job with the new values, but - preserves `run_at`. This is primarily useful for **throttling** (executing at - most once over a given time period). Locked jobs will cause a new job to be - scheduled instead. +- `replace` (default) - overwrites the unlocked job with the new values (merging + array payloads). This is primarily useful for rescheduling, updating, or + **debouncing** (delaying execution until there have been no events for at + least a certain time period). Locked jobs will cause a new job to be scheduled + instead. +- `preserve_run_at` - overwrites the unlocked job with the new values (merging + array payloads), but preserves `run_at`. This is primarily useful for + **throttling** (executing at most once over a given time period). Locked jobs + will cause a new job to be scheduled instead. - `unsafe_dedupe` - if an existing job is found, even if it is locked or permanently failed, then it won't be updated. This is very dangerous as it means that the event that triggered this `add_job` call may not result in any @@ -73,10 +85,12 @@ The full `job_key_mode` algorithm is roughly as follows: ### Array payload merging -When updating an existing job via `job_key`, if both the existing job's -payload and the new payload are JSON arrays, they will be **concatenated** -rather than replaced. This enables a batching pattern where multiple -events can be accumulated into a single job (except in `unsafe_dedupe` mode). +When updating an existing job via `job_key` (except in `unsafe_dedupe` mode), if +both the existing job's payload and the new payload are JSON arrays, they will +be concatenated rather than overwritten. This enables a batching pattern where +multiple events can be accumulated into a single job for more efficient +execution; see [Handling batch jobs](./tasks.md#handling-batch-jobs) for more +info. ```sql -- First call creates job with payload: [{"id": 1}] @@ -98,17 +112,17 @@ SELECT graphile_worker.add_job( ); ``` -Combined with `preserve_run_at` job_key_mode, this creates a fixed batching window: the job -runs at the originally scheduled time with all accumulated payloads merged -together. With the default `replace` job_key_mode, each new event would push the -`run_at` forward, creating a rolling/debounce window instead. +Combined with `preserve_run_at` job_key_mode, this creates a fixed batching +window: the job runs at the originally scheduled time with all accumulated +payloads merged together. With the default `replace` job_key_mode, each new +event would push the `run_at` forward, creating a rolling/debounce window +instead. -:::caution +:::caution[Both payloads must be arrays for merging to occur] -If **either** payload is not an array (e.g., one is an object), the standard -replace behavior applies and the old payload will be lost. Note that the -default payload is an object, so calling `add_job(s)` without an explicit -payload will also trigger a replace rather than a concatenation. +If **either** payload is not an array (e.g., one is an object, as is the default +if no payload is specified), the standard replace behavior applies and the old +payload will be lost. ::: @@ -139,3 +153,9 @@ prevented from running again, and will have the `job_key` removed from it.) Calling `remove_job` for a locked (i.e. running) job will not actually remove it, but will prevent it from running again on failure. + +There's currently a race condition in adding jobs with a job key which means +under very high contention of a specific key an `add_job` may fail and return +`null`. You should check for this `null` and handle it appropriately: retrying, +throwing an error, or however else makes sense to your code. See: +https://github.com/graphile/worker/issues/580 for more details. diff --git a/website/docs/tasks.md b/website/docs/tasks.md index 7c350d78..9a2faf05 100644 --- a/website/docs/tasks.md +++ b/website/docs/tasks.md @@ -193,11 +193,12 @@ any of these promises reject, then the job will be re-enqueued, but the payload will be overwritten to only contain the entries associated with the rejected promises — i.e. the successful entries will be removed. -:::tip Accumulating batch payloads with job_key +:::tip[Accumulating batch payloads with job_key] You can use [`job_key`](./job-key.md#array-payload-merging) with array payloads to accumulate multiple events into a single batch job. When both the existing -and new payloads are arrays, they are concatenated automatically. +and new payloads are arrays (and `job_key_mode` is not `unsafe_dedupe`), they +are concatenated automatically. ::: From 860ece9d9fb513c26276a79dbd6233bb4bac3651 Mon Sep 17 00:00:00 2001 From: Benjie Gillam Date: Fri, 27 Feb 2026 10:46:17 +0000 Subject: [PATCH 4/4] We're running an older Docusaurus --- website/docs/job-key.md | 2 +- website/docs/tasks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/job-key.md b/website/docs/job-key.md index 1693fb36..75805197 100644 --- a/website/docs/job-key.md +++ b/website/docs/job-key.md @@ -118,7 +118,7 @@ payloads merged together. With the default `replace` job_key_mode, each new event would push the `run_at` forward, creating a rolling/debounce window instead. -:::caution[Both payloads must be arrays for merging to occur] +:::caution Both payloads must be arrays for merging to occur If **either** payload is not an array (e.g., one is an object, as is the default if no payload is specified), the standard replace behavior applies and the old diff --git a/website/docs/tasks.md b/website/docs/tasks.md index 9a2faf05..4408f849 100644 --- a/website/docs/tasks.md +++ b/website/docs/tasks.md @@ -193,7 +193,7 @@ any of these promises reject, then the job will be re-enqueued, but the payload will be overwritten to only contain the entries associated with the rejected promises — i.e. the successful entries will be removed. -:::tip[Accumulating batch payloads with job_key] +:::tip Accumulating batch payloads with job_key You can use [`job_key`](./job-key.md#array-payload-merging) with array payloads to accumulate multiple events into a single batch job. When both the existing