-
Notifications
You must be signed in to change notification settings - Fork 10
Description
Problem
OrchestrationExecutor.handleExecutionResumed() in packages/durabletask-js/src/worker/orchestration-executor.ts iterates _suspendedEvents using for..of while processEvent() can push new entries onto the same array during iteration.
If the suspended-event buffer contains an ExecutionSuspended event (e.g. from a double-suspend sequence), processing it sets _isSuspended = true. Any subsequent suspendable event in the buffer is then re-pushed onto the same array being iterated, and JavaScript's for..of on arrays iterates dynamically-added elements — creating an infinite loop that hangs the worker process.
Before the fix:
// orchestration-executor.ts — handleExecutionResumed
this._isSuspended = false;
for (const e of this._suspendedEvents) { // ← iterates the live array
await this.processEvent(ctx, e); // ← can push back to the same array
}
this._suspendedEvents = []; // ← never reached during infinite loopAdditionally, isSuspendable() in worker/index.ts does not exclude EXECUTIONSUSPENDED, allowing suspend events to be buffered. It also uses loose equality (== -1) instead of strict equality (=== -1).
Root Cause
for..ofon a JavaScript array uses an iterator that checkslengthdynamically. Pushing to the array during iteration causes the new elements to be iterated, enabling the infinite loop.isSuspendable()does not excludeEXECUTIONSUSPENDEDevents, so a second suspend event received while already suspended gets buffered rather than processed immediately (as a no-op).
Proposed Fix
- Snapshot the buffer before iterating: Assign
_suspendedEventsto a local variable and replace it with an empty array before the loop. Events buffered during processing are correctly held for the next Resume. - Exclude
EXECUTIONSUSPENDEDfromisSuspendable()so suspend events are never buffered (defense-in-depth). - Use strict equality (
===) in theindexOfcheck.
Impact
- Severity: High — can hang the worker process indefinitely, causing the orchestration to never complete.
- Affected scenarios: Any orchestration that receives duplicate or back-to-back
ExecutionSuspendedevents from the sidecar, or during history replay of such sequences. - Likelihood: Low in normal operation (the sidecar typically does not send consecutive suspend events), but the code should be defensive against it.