Skip to content

docs(flaky-tests): document automatic label application from monitors #651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
samgutentag wants to merge 1 commit into from
Closed

docs(flaky-tests): document automatic label application from monitors #651

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions flaky-tests/management/test-labels.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Organize and categorize test cases with organization-scoped labels.

# Test Labels

Test labels are organization-scoped tags you can apply to individual test cases to organize, filter, and categorize your test suite. Labels are applied manually today; see [Automatic labeling from monitors](#automatic-labeling-from-monitors) for what's coming.
Test labels are organization-scoped tags you can apply to individual test cases to organize, filter, and categorize your test suite. Labels can be applied manually or automatically by monitors.

<figure><img src="../../.gitbook/assets/test-details-labels.png" alt="Labels applied to a test on details page"><figcaption></figcaption></figure>

Expand Down Expand Up @@ -32,9 +32,24 @@ On the tests list, you can filter the table down to test cases that have a parti

### Automatic labeling from monitors

{% hint style="info" %}
**Coming soon.** Monitors will be able to automatically apply and remove labels on test cases based on test behavior. More details will be published when this is available.
{% endhint %}
Monitors can automatically apply labels to test cases when the monitor activates. This lets you tag tests as they accumulate failures without any manual triage.

When creating a failure count or failure rate monitor, you choose the **action type**:

* **Classify test status** — the monitor marks tests as flaky or broken (the default behavior)
Copy link
Copy Markdown
Contributor

@claude claude Bot May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: "(the default behavior)" — since the parallel bullet Apply labels also describes a behavior, "default behavior" reads slightly awkwardly. Consider just "(default)" for parallelism, e.g.:

Suggested change
* **Classify test status** — the monitor marks tests as flaky or broken (the default behavior)
* **Classify test status** — the monitor marks tests as flaky or broken (default)

Non-blocking.

* **Apply labels** — the monitor tags matching tests with one or more labels when it activates

The action type is set at creation and cannot be changed afterward. If you need to switch a monitor's action type, create a new monitor with the desired type and disable the old one.

To configure automatic labeling:

1. Go to **Flaky Tests** > select your repository > **Monitors**
2. Click **Add** next to the monitor type you want to create
3. In the creation form, select **Apply labels** as the action type
4. Choose one or more labels to apply when the monitor activates
5. Complete the rest of the monitor configuration and save
Comment on lines +46 to +50
Copy link
Copy Markdown
Contributor

@claude claude Bot May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style consistency — numbered list punctuation.

Other numbered lists in this directory (e.g., flaky-tests/management/managing-detected-flaky-tests.md and flaky-tests/detection/failure-count-monitor.md) end each step with a period. These steps don't, which makes the list look inconsistent with the rest of the docs.

Suggested change
1. Go to **Flaky Tests** > select your repository > **Monitors**
2. Click **Add** next to the monitor type you want to create
3. In the creation form, select **Apply labels** as the action type
4. Choose one or more labels to apply when the monitor activates
5. Complete the rest of the monitor configuration and save
1. Go to **Flaky Tests** > select your repository > **Monitors**.
2. Click **Add** next to the monitor type you want to create.
3. In the creation form, select **Apply labels** as the action type.
4. Choose one or more labels to apply when the monitor activates.
5. Complete the rest of the monitor configuration and save.

Fix this →


Tests that the monitor flags will have the selected labels applied automatically. You can still apply or remove labels manually on top of what monitors set.
Copy link
Copy Markdown
Contributor

@claude claude Bot May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional wording polish: "what monitors set" is a bit terse and uses passive voice. Consider:

Suggested change
Tests that the monitor flags will have the selected labels applied automatically. You can still apply or remove labels manually on top of what monitors set.
When the monitor flags a test, the selected labels are applied automatically. You can still apply or remove labels manually on top of what monitors apply.

Or keep the passive but use "apply" for parallelism with the rest of the section: "on top of what monitors apply" rather than "set". Non-blocking.


### Related

Expand Down
Loading