Skip to content

Aggregate internal task queue metrics under __temporal_sys__ tag#9769

Closed
rkannan82 wants to merge 8 commits intomainfrom
kannan/suppress-internal-tq-metrics
Closed

Aggregate internal task queue metrics under __temporal_sys__ tag#9769
rkannan82 wants to merge 8 commits intomainfrom
kannan/suppress-internal-tq-metrics

Conversation

@rkannan82
Copy link
Copy Markdown
Contributor

@rkannan82 rkannan82 commented Apr 1, 2026

What

In GetPerTaskQueueFamilyScope, task queues with the /temporal-sys/ prefix are automatically tagged as taskqueue=__temporal_sys__ instead of using the actual queue name, regardless of BreakdownMetricsByTaskQueue setting.

Why

Internal task queues (e.g. /temporal-sys/worker-commands/{namespace}/{key}) are per-worker and would cause cardinality explosion. Aggregating under a synthetic label preserves observability while bounding cardinality — same pattern as sticky queues using __sticky__.

How did you test it?

Unit test in matching_engine_test.go that emits a metric through the per-partition handler and asserts the taskqueue tag value.

@rkannan82 rkannan82 changed the base branch from main to krajah/nexus-client-name-metric April 1, 2026 18:48
@rkannan82 rkannan82 marked this pull request as ready for review April 7, 2026 00:31
@rkannan82 rkannan82 requested a review from a team as a code owner April 7, 2026 00:31
@rkannan82 rkannan82 requested a review from ShahabT April 7, 2026 00:31
@rkannan82 rkannan82 force-pushed the kannan/suppress-internal-tq-metrics branch 2 times, most recently from 6498271 to c43a8c3 Compare April 7, 2026 00:52
@rkannan82 rkannan82 force-pushed the kannan/suppress-internal-tq-metrics branch from c43a8c3 to e26027a Compare April 7, 2026 05:23
@rkannan82 rkannan82 requested review from a team as code owners April 7, 2026 05:23
@rkannan82 rkannan82 changed the title Suppress per-task-queue metrics for /temporal-sys/ internal queues Aggregate internal task queue metrics under __temporal_sys__ tag Apr 7, 2026
Base automatically changed from krajah/nexus-client-name-metric to main April 7, 2026 05:34
rkannan82 and others added 4 commits April 6, 2026 22:35
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… handler

Instead of suppressing all metrics for /temporal-sys/ queues with a noop
handler, aggregate them under a synthetic "__temporal_sys__" task queue tag.
This preserves operational observability while preventing cardinality explosion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/suppress-internal-tq-metrics branch from e2d762d to f127143 Compare April 7, 2026 05:36
@rkannan82 rkannan82 requested a review from ShahabT April 7, 2026 05:38
@rkannan82 rkannan82 enabled auto-merge (squash) April 7, 2026 05:38
Addresses review feedback: the prefix check now lives in the metrics
package so all callers of GetPerTaskQueue*Scope automatically get the
__temporal_sys__ aggregation, not just loggerAndMetricsForPartition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/suppress-internal-tq-metrics branch from 2958e84 to 4353eb9 Compare April 7, 2026 21:42
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/suppress-internal-tq-metrics branch from 4353eb9 to cb8e05b Compare April 7, 2026 21:44
@rkannan82 rkannan82 requested a review from ShahabT April 7, 2026 21:45
@rkannan82
Copy link
Copy Markdown
Contributor Author

Closing this PR in favor of introducing a new task queue kind.
See
temporalio/api#763 and #9899

@rkannan82 rkannan82 closed this Apr 10, 2026
auto-merge was automatically disabled April 10, 2026 04:43

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants