Skip to content

Add WorkerCommandsTask outbound task to dispatch worker commands via Nexus#9232

Merged
rkannan82 merged 46 commits intomainfrom
kannan/activity-cancel/task-definition
Apr 11, 2026
Merged

Add WorkerCommandsTask outbound task to dispatch worker commands via Nexus#9232
rkannan82 merged 46 commits intomainfrom
kannan/activity-cancel/task-definition

Conversation

@rkannan82
Copy link
Copy Markdown
Contributor

@rkannan82 rkannan82 commented Feb 5, 2026

What changed?

New outbound task type (WorkerCommandsTask) that carries worker commands to be dispatched to workers via Nexus. Uses the generic WorkerCommand proto (not cancel-activity-specific), so this task type can carry any future command types.

Suggested review order: proto changes → worker_commands_task.gotask_generator.goworkflow_task_completed_handler.go

Key pieces:

  • Proto: TASK_TYPE_WORKER_COMMANDS enum, WorkerCommandsTask in OutboundTaskInfo with repeated WorkerCommand.
  • Task definition: worker_commands_task.go — implements outbound Task and HasDestination interfaces.
  • Task creation (workflow_task_completed_handler.go, task_generator.go): When RequestCancelActivityTask is processed for a started activity whose worker has a control queue, collects a CancelActivityCommand with the activity's task token. Commands are batched by destination control queue and flushed as one WorkerCommandsTask per queue at the end of WFT processing.
  • Serialization: task_serializers.go for persistence round-tripping.

Dispatch is a no-op here — handled in #9233. Gated by dynamic config EnableCancelActivityWorkerCommand (default: off).

Why?

To support proactive activity cancellation without waiting for heartbeat. This is the task creation leg of the flow.

  1. [Store worker attributes needed by server to propagate nexus tasks to worker #9231] Store worker_control_task_queue in ActivityInfo at activity start.
  2. [This PR] On RequestCancelActivityTask, batch commands by control queue into WorkerCommandsTask outbound tasks.
  3. [Dispatch activity cancellation to worker using Nexus #9233] Dispatch each task as a Nexus ExecuteCommands operation to the worker, with a 3-attempt retry cap.
  4. [SDK] Worker receives the cancel command and cancels the running activity.

Gated by dynamic config EnableCancelActivityWorkerCommand (default: off).

How did you test it?

Unit tests cover task generation, command batching (including multi-queue batching), task serialization round-tripping, and the feature-flag-off path.

@rkannan82 rkannan82 requested review from a team as code owners February 5, 2026 20:00
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 3 times, most recently from 9eea93d to 835d5bf Compare February 5, 2026 21:10
@rkannan82 rkannan82 requested a review from yycptt February 5, 2026 21:11
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from d402868 to 54b6d1a Compare February 6, 2026 00:36
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from 5bae03a to a5e43dd Compare February 11, 2026 19:48
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 2 times, most recently from 417606c to 6246c4e Compare February 11, 2026 21:04
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from f1e1ee5 to e795849 Compare February 11, 2026 21:07
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 2 times, most recently from fec3c41 to 1dd975d Compare February 11, 2026 21:22
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 11c74b6 to 2cb3108 Compare February 12, 2026 00:43
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from d0f60d7 to fa4be0a Compare February 12, 2026 06:53
@rkannan82 rkannan82 requested a review from yycptt February 12, 2026 07:04
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 2 times, most recently from 89d2cf5 to 79e4e3e Compare February 15, 2026 03:33
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from bf3c061 to 1218cae Compare February 15, 2026 03:42
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 3 times, most recently from 9231795 to bc58c4f Compare February 16, 2026 02:44
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from 04db6f1 to b72cf9f Compare February 16, 2026 02:48
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from bc58c4f to 3eb028c Compare February 16, 2026 02:48
@rkannan82 rkannan82 changed the title Add new transfer task type for cancelling activities using Nexus [DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities Feb 16, 2026
@rkannan82 rkannan82 requested review from a team as code owners April 7, 2026 19:09
rkannan82 and others added 2 commits April 7, 2026 12:10
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 7040cec to 0b36762 Compare April 9, 2026 21:59
rkannan82 and others added 4 commits April 9, 2026 15:00
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Main already has the direct dependency at the same version.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts:
- task.proto: bump TASK_TYPE_WORKER_COMMANDS from 34 to 35 (34 taken by REPLICATION_DELETE_EXECUTION)
- executions.proto: merge imports, keep both outbound task details and worker_control_task_queue field
- mutable_state_impl_test.go: keep both new test functions
- go.mod: update go.temporal.io/api to v1.62.8-0.20260407190616-8574d6aa8b01
- Regenerated .pb.go files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 requested a review from a team as a code owner April 9, 2026 23:05
@semgrep-managed-scans
Copy link
Copy Markdown

Semgrep found 1 missing-explicit-permissions finding:

  • .github/workflows/promote-docker-image.yml

No explicit GITHUB_TOKEN permissions found at the workflow or job level. Add a permissions: block at the workflow root (applies to all jobs) or per job with least privilege (e.g., contents: read and only specific writes like pull-requests: write if needed).

…orker-key' into kannan/activity-cancel/task-definition

# Conflicts:
#	api/enums/v1/task.go-helpers.pb.go
#	api/enums/v1/task.pb.go
#	api/persistence/v1/executions.pb.go
#	go.mod
#	go.sum
#	proto/internal/temporal/server/api/enums/v1/task.proto
#	proto/internal/temporal/server/api/persistence/v1/executions.proto
#	service/matching/matching_engine_test.go
#	service/matching/task_queue_partition_manager.go
#	tests/query_workflow_test.go
@rkannan82 rkannan82 requested a review from yycptt April 10, 2026 01:47
rkannan82 added a commit that referenced this pull request Apr 11, 2026
…worker (#9231)

## What changed?
As part of RecordActivityTaskStarted flow, store
worker_control_task_queue for an activity in the mutable state
(ActivityInfo).

Main changes:
- executions.proto: Added the new worker_control_task_queue field.
- mutable_state_impl.go: Update mutable state.
- matching/forwarder.go: Propagate worker_control_task_queue when polls
get forwarded. Otherwise, RecordActivityTaskStarted request will not
have it set when invoked from a forwarded poll.

## Why?
To support activity cancellation without activity heartbeat.

Overall flow:
- [This PR] Store worker attributes in ActivityInfo as part of
RecordActivityTaskStarted call.
- [#9232] When user cancels a workflow, create 1 or more tasks. Group
all activities belonging to a worker into the task (for efficiency).
- [#9233] Lookup the Nexus task queue for each worker, and send a Nexus
operation for each transfer task.
- [SDK] Worker will receive this cancel task and cancel the running
activities.

## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Base automatically changed from kannan/activity-cancel/persist-worker-key to main April 11, 2026 00:51
rkannan82 and others added 6 commits April 10, 2026 18:18
…l/task-definition

# Conflicts:
#	go.mod
#	go.sum
#	service/history/api/respondworkflowtaskcompleted/workflow_task_completed_handler_test.go
#	tests/query_workflow_test.go
The pre-release version was missing ContinueAsNewInitialVersioningBehavior
fields that landed in the released tag. The released v1.62.8 includes both
our WorkerCommand proto and the versioning fields from main.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reset matching_engine_test.go, task_queue_partition_manager.go, and
protogen to match main — these were pulled in by duplicate merge commits
and are not part of this PR's changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 enabled auto-merge (squash) April 11, 2026 02:52
@rkannan82 rkannan82 disabled auto-merge April 11, 2026 02:52
@rkannan82 rkannan82 merged commit f5246b3 into main Apr 11, 2026
70 checks passed
@rkannan82 rkannan82 deleted the kannan/activity-cancel/task-definition branch April 11, 2026 03:48
rkannan82 added a commit that referenced this pull request Apr 11, 2026
Resolves conflicts after #9924 (Executable.Attempt) and #9232
(task-definition) merged to main. Regenerated proto files for
started_clock field addition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants