Activity task can be leased and marked running before Python worker receives usable poll response

[rmcdaniel]

Summary

On a clean published-artifact polyglot stack, a PHP-authored workflow can schedule a Python activity, the server can durably lease that activity task and mark the execution/attempt running, but the Python worker never surfaces a usable activity poll response and never reaches user activity code.

This looks related to, but distinct from, #4:

• Resumed Python workflow task can be leased but not delivered before default poll timeout #4 is the resumed workflow-task path
• this issue is the activity-task path

Reproduction

Environment used:

• durableworkflow/server:0.2.109
• durable-workflow/workflow 2.0.0-alpha.143
• durable-workflow==0.4.33
• shared queue polyglot-shared
• one PHP workflow worker, one PHP activity worker, one Python worker

Failing case:

1. Start the published PHP workflow worker and published Python worker on the same task queue.
2. Start workflow polyglot.php.calls_python.
3. The PHP workflow worker handles the initial workflow task and schedules polyglot.activity.python.echo.
4. The client waits for completion with handle.result(timeout=240).
5. The run stays waiting until timeout.

Worker evidence

• PHP workflow worker log:
  • php workflow worker handled polyglot.php.calls_python with 1 command(s)
• Python worker log:
  • registered successfully
  • kept heartbeating
  • never logged python activity echo received payload type=dict
  • never logged a successful activity-task completion for the run

Durable state from MySQL

Stalled run php_calls_python-584c62aa:

• workflow run 01krmhz6vcc0s4chwc7ad8ktac: status=waiting
• initial workflow task 01krmhz6w49k76v51n01m4cbp3: task_type=workflow, status=completed, lease_owner=php-workflow-alpha143-20260515
• follow-up task 01krmhz73var2n92zm535200kd: task_type=activity, status=leased, lease_owner=py-worker-alpha143-20260515
• activity execution 01krmhz73qww598x54geyyy910: activity_type=polyglot.activity.python.echo, status=running
• activity attempt 01KRMHZS3RB9H4AGR4RT1ZZHAT: status=running, lease_owner=py-worker-alpha143-20260515

So the server has already moved durable activity state forward to running, but the Python worker never surfaces usable activity execution on its side.

Expected

If the activity poll response is not durably usable by the worker, the server should not strand the activity in a leased/running state with no replay path.

Actual

The activity task can become leased and the execution/attempt can become running before the Python worker ever logs activity start or completion.

[rmcdaniel]

Merged the server-side activity poll replay fix in #6.

Also shipped the supporting Python client change as durable-workflow 0.4.35, which now sends a stable poll_request_id on activity polls and retries one timeout with the same id.

The remaining validation step is a published-artifact rerun against durableworkflow/server:0.2.110 once the Docker release workflow finishes pushing the tag.

[rmcdaniel]

Merged the server-side activity poll replay fix in #6.

Also shipped the supporting Python client change as durable-workflow 0.4.35, which now sends a stable poll_request_id on activity polls and retries one timeout with the same id.

The remaining validation step is a published-artifact rerun against durableworkflow/server:0.2.110 once the Docker release workflow finishes pushing the tag.

[rmcdaniel]

Published validation passed on shipped artifacts.

Pinned versions:

• server: durableworkflow/server:0.2.110
• cli: 0.1.37
• sdk-python: durable-workflow 0.4.35
• workflow: durable-workflow/workflow 2.0.0-alpha.143
• waterline: durable-workflow/waterline 2.0.0-alpha.47

Four-corner shared-queue smoke passed:

• PHP workflow -> Python activity
• Python workflow -> PHP activity
• PHP workflow -> PHP activity
• Python workflow -> Python activity

One environment note from the validation harness: inside this workspace container, the host-published Docker port was reachable at 172.24.0.1 rather than 127.0.0.1. That was a harness networking detail, not a product regression.