sleep() is not durable across worker crashes in Postgres World

[abdullah-hallaq]

Description

Using Workflow DevKit with Postgres World, I’m observing non-durable behavior for sleep() when the worker/server is down at the moment the sleep duration expires.

According to the docs:

• sleep: Suspends a workflow for a specified duration or until an end date without consuming any resources. Once the duration or end date passes, the workflow will resume execution.
• Postgres World is described as a long‑running, persistent world using PostgreSQL + pg-boss.

Based on this, I expect sleep() to be durable across process crashes, even if the worker is offline at the exact wake-up time.

Observed behavior:

• If the worker comes back before the sleep duration elapses, the workflow resumes as expected.
• If the worker comes back after the sleep duration has already passed, the workflow never resumes and the run remains stuck in a running state.

---

Environment

• World: postgres-world
• Worker: running with environment variables per docs:
  • WORKFLOW_TARGET_WORLD=postgres-world
  • WORKFLOW_POSTGRES_URL=<same for app and worker>
  • WORKFLOW_POSTGRES_WORKER_CONCURRENCY set (default / reasonable value)
• The worker and app both point to the same PostgreSQL instance

---

Reproduction steps

Workflow Test Scenario

1. Log: "before sleep"
2. sleep(5000) (5 seconds)
3. Log: "after sleep"

Scenario A (works as expected)

• [10:00] Start workflow; it calls sleep(5s)
• [10:01] Crash/stop the server/worker
• [10:04] Restart the server/worker (before 5s elapsed)
• [10:05] Workflow resumes and logs "after sleep"

Scenario B (buggy behavior)

• [10:00] Start workflow; it calls sleep(5s)
• [10:01] Crash/stop the server/worker
• [10:05+] Restart the server/worker (after 5s elapsed)

Result:

• Workflow does not resume
• Run remains stuck in running state in the database
• No further steps execute

The only difference between scenarios A and B is whether the worker is alive at the wake-up time.

Minimal workflow:

import { sleep } from "workflow";

export async function handleUserSignup() {
  "use workflow";

  await logStarted();

  await sleep(2000);
  await log1();

  await sleep(2000);
  await log2();

  return { status: "success" };
}

async function logStarted() {
  "use step";
  console.log("Started");
}

async function log1() {
  "use step";
  console.log("Log 1 - Stop here to test crash recovery");
}

async function log2() {
  "use step";
  console.log("Log 2 - Success!");
}

[pranaygp]

I'm unsure how pg-boss is meant to handle workers being down during a message wakeup time. perhaps we need to implement a check for 'expired' queue messages in workers/the world setup on wake up? and a world test for this? cc @Schniz

[abdullah-hallaq]

Hello,
Just following up - any update on this? @pranaygp @Schniz

[seekcx]

Same problem. I observed this phenomenon in the local world and the same in the PG world. If I click "Cancel active sleeps" in the UI, the task will be executed immediately.

In contrast, even if I restart the service before the sleep expires, it will not trigger the subsequent steps, but will remain stuck in the sleep stage.

I use PG world, This is my demo:

import { sleep } from "workflow";

async function step1(value: number) {
  "use step";
  console.log("Executing Step 1: ", value);
  return value + 1;
}

async function step2(value: number) {
  "use step";

  console.log("Executing Step 2: ", value);

  return value * 2;
}

async function demo(value: number) {
  "use workflow";

  const newValue = await step1(value);

  await sleep("10s");
  const finalValue = await step2(newValue);
  console.log(`Final Value: ${finalValue}`);
}

[jcourson-bg]

@pranaygp I've been running into a similar issue and spent some time tracing through the source. I think the problem is broader than just sleep(). Looks like it could affect all execution recovery in postgres world.

When graphile-worker picks up a job, the task handler in world-postgres/src/queue.ts does:

await localWorld.queue(queueName, message, { ... });

But if you look at world-local/src/queue.ts, that queue() call kicks off execution in a detached (async () => { ... })() and returns { messageId } immediately at line 202 - the IIFE is never awaited. So from graphile-worker's perspective the job completed successfully, even though the actual workflow execution just started in the background.

If the process dies while that detached execution is running, there's no graphile-worker job left to retry. The event log is still there in postgres, but nothing re-triggers the workflow.

In world-vercel/src/queue.ts, the handler is awaited directly inside the VQS callback:

const result = await handler(payload, { ... });

And there's even a comment at line 193 about crash safety:

// Send new message BEFORE acknowledging current message.
// This ensures crash safety: if process dies after send but before ack,
// we may get a duplicate invocation but won't lose the scheduled wakeup.

I wonder if the fix is to do something similar in postgres world? Skip the localWorld delegation entirely and just await the handler from within the graphile-worker task? That way the job stays locked to the worker's pg session for the full duration of execution, and graphile-worker's built-in retry handles the crash case. For timeoutSeconds (sleep), you could maybe re-enqueue with graphile-worker's runAt instead of relying on the local world's setTimeout.

Not sure if the local world indirection is intentional for composability (the comment at line 43 hints at that), but it seems like it breaks the durability guarantee postgres world is supposed to provide.

[pranaygp]

@jcourson-bg I think you're right and we should ideally just rely on graphile-worker's internal retrying mechanism. Forking off the the local world queue was an initial implementation and something we should deprecate in favor of this. Looks like you're working on a PR already?

[jcourson-bg]

@pranaygp that PR was premature - removed it, but I'll give a stab at it.

[jcourson-bg]

@pranaygp Open-source graphile-worker has a hardcoded 4-hour stale lock timeout (resetLockedAt). Worker Pro solves this with a heartbeat/sweep system, but it's $25-100/mo. Since graphile exposes forceUnlockWorkers() in the open-source API, I think we could build the same mechanism ourselves:

A workflow.worker_heartbeats table tracking each runner's pool ID and last_seen timestamp
A periodic heartbeat (default 30s) that touches the row while the process is alive
On startup + periodically: sweep for pools whose heartbeat has gone stale, call forceUnlockWorkers() with their worker IDs

This seems safe for multi-container because it only releases locks from workers whose heartbeat is stale (not a time-based guess about lock age), so long-running tasks on healthy containers aren't affected.

What do you think about this approach?