Resumed Python workflow task can be leased but not delivered before default poll timeout

[rmcdaniel]

Summary

On a clean published-artifact polyglot stack, a resumed Python-authored workflow task can be created and leased after a Python activity completes, but the polling Python worker never receives a usable poll response and the workflow stays waiting until client result timeout.

Reproduction

Environment used:

• durableworkflow/server:0.2.109
• durable-workflow==0.4.33
• shared queue polyglot-shared
• one Python worker registered for both workflows and activities

Minimal failing case:

1. Start a Python worker with the default DURABLE_WORKFLOW_POLL_TIMEOUT_SECONDS=30.
2. Start workflow polyglot.python.calls_python with a simple dict payload.
3. The Python worker completes the initial workflow task and schedules polyglot.activity.python.echo.
4. The Python worker completes that activity successfully.
5. The server creates a follow-up workflow task for the same run, but the workflow remains stuck in waiting and handle.result(timeout=240) times out.

Clean-stack evidence from the failing run:

• workflow run python_calls_python-177b48f7 stayed waiting
• initial workflow task 01krmce8ndt6yw1n0ryzc2yr4g completed successfully
• activity execution 01krmce92e36datpbtfdvp2zcn completed successfully
• follow-up workflow task 01krmcf7xaa4efkd2cj00gys90 was created and then leased to the Python worker
• the worker never logged a successful workflow-tasks/poll response or workflow re-entry for that follow-up task

Relevant durable state from MySQL for the failing run:

• run status: waiting
• follow-up task row: 01krmcf7xaa4efkd2cj00gys90, status=leased, lease_owner=py-worker-4e478e0100d9-13
• task payload was small metadata only:
  • {"open_wait_id":"activity:01krmce92e36datpbtfdvp2zcn","activity_type":"polyglot.activity.python.echo",...}

Expected

Once the Python activity completes, the follow-up workflow task should be returned cleanly through worker poll and the workflow should complete.

Actual

The follow-up task is created and leased, but the worker never gets a usable poll response for it, and the workflow remains stuck.

Workaround

Raising the Python worker poll timeout from 30 to 60 made both a targeted python_calls_python repro and the full four-corner polyglot smoke pass on the same published artifacts.

That suggests either:

• the server is leasing the resumed task before the worker poll response is fully deliverable, or
• the resumed-task poll response path is slow enough that the default Python poll timeout is too tight.

[rmcdaniel]

Fresh published-artifact rerun after shipping durable-workflow/workflow 2.0.0-alpha.143 fixed the original PHP worker registration blocker, but the stack then advanced to a later stall that looks like the same lease-without-usable-delivery class.

Pinned versions in this rerun:

• durableworkflow/server:0.2.109
• durable-workflow/workflow 2.0.0-alpha.143
• durable-workflow==0.4.33
• shared queue polyglot-shared

Observed failing case:

• start polyglot.php.calls_python
• PHP workflow worker registers and handles the initial workflow task
• Python worker is registered and heartbeating
• client handle.result(timeout=240) times out with the run still waiting

Relevant worker evidence:

• PHP workflow worker log: php workflow worker handled polyglot.php.calls_python with 1 command(s)
• Python worker never logged python activity echo received payload type=dict
• Python worker never logged a successful activity-task completion for that run

Relevant durable DB state for the stalled run php_calls_python-584c62aa:

• run 01krmhz6vcc0s4chwc7ad8ktac: status=waiting
• initial workflow task 01krmhz6w49k76v51n01m4cbp3: task_type=workflow, status=completed, lease_owner=php-workflow-alpha143-20260515
• follow-up task 01krmhz73var2n92zm535200kd: task_type=activity, status=leased, lease_owner=py-worker-alpha143-20260515
• activity execution 01krmhz73qww598x54geyyy910: activity_type=polyglot.activity.python.echo, status=running
• activity attempt 01KRMHZS3RB9H4AGR4RT1ZZHAT: status=running, lease_owner=py-worker-alpha143-20260515, lease_expires_at=2026-05-15 00:58:36

So on this run the server advanced past registration and workflow scheduling, then durably marked the Python activity as running, but the worker never surfaced usable activity execution on its side.

That looks consistent with the same general class as this issue: the server can move durable lease state forward before the Python worker gets a usable poll result, except here it is happening on the activity path rather than only the resumed workflow-task path.

[rmcdaniel]

Follow-through on the workflow-task side of this issue is now shipped in the Python SDK:

• source PR: Retry workflow polls with stable request ids sdk-python#1
• release: https://github.com/durable-workflow/sdk-python/releases/tag/0.4.34
• PyPI: durable-workflow==0.4.34

That change starts sending a poll_request_id on workflow task polls and retries one timed-out workflow poll once with the same request id so the server can replay a cached leased workflow task instead of dropping it.

The separate activity-path stall from the fresh polyglot rerun still remains to be addressed.

[rmcdaniel]

Fixed by the shipped workflow-poll request-id replay path and validated on published artifacts.

Published rerun evidence:

• server: durableworkflow/server:0.2.110
• sdk-python: durable-workflow 0.4.35
• cli: 0.1.37
• workflow: durable-workflow/workflow 2.0.0-alpha.143
• waterline: durable-workflow/waterline 2.0.0-alpha.47

The previously failing python_calls_python path now completes at the default worker poll timeout on the published stack, and the full four-corner shared-queue smoke passes on the same artifacts.

One harness-only note from the validation environment: inside this workspace container, the host-published Docker port was reachable at 172.24.0.1 rather than 127.0.0.1. That was not a product regression.