Skip to content

Sync content from trunk-io/docs #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mintlify wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Sync content from trunk-io/docs #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions docs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@
"merge-queue/using-the-queue/monitor-queue-status",
"merge-queue/using-the-queue/handle-failed-pull-requests",
"merge-queue/using-the-queue/stacked-pull-requests",
"merge-queue/using-the-queue/emergency-pull-requests"
"merge-queue/using-the-queue/emergency-pull-requests",
"merge-queue/using-the-queue/force-merge"
]
},
{
Expand All @@ -111,7 +112,8 @@
"pages": [
"merge-queue/administration",
"merge-queue/administration/advanced-settings",
"merge-queue/administration/metrics"
"merge-queue/administration/metrics",
"merge-queue/administration/terraform"
]
},
{
Expand Down Expand Up @@ -198,8 +200,12 @@
"pages": [
"flaky-tests/detection",
"flaky-tests/detection/pass-on-retry-monitor",
"flaky-tests/detection/threshold-monitor",
"flaky-tests/detection/failure-rate-monitor",
"flaky-tests/detection/failure-count-monitor",
"flaky-tests/detection/flag-as-flaky",
"flaky-tests/test-labels",
"flaky-tests/autofix-flaky-tests",
"flaky-tests/autofix-ci-failures",
"flaky-tests/infrastructure-failure-protection",
"flaky-tests/the-importance-of-pr-test-results",
"flaky-tests/quarantining",
Expand Down Expand Up @@ -239,8 +245,11 @@
"flaky-tests/use-mcp-server/configuration/github-copilot-ide",
"flaky-tests/use-mcp-server/configuration/claude-code-cli",
"flaky-tests/use-mcp-server/configuration/gemini-cli",
"flaky-tests/use-mcp-server/configuration/bearer-auth",
"flaky-tests/use-mcp-server/mcp-tool-reference",
"flaky-tests/use-mcp-server/mcp-tool-reference/get-root-cause-analysis",
"flaky-tests/use-mcp-server/mcp-tool-reference/fix-flaky-test",
"flaky-tests/use-mcp-server/mcp-tool-reference/investigate-ci-failure",
"flaky-tests/use-mcp-server/mcp-tool-reference/search-test",
"flaky-tests/use-mcp-server/mcp-tool-reference/set-up-test-uploads"
]
}
Expand All @@ -258,6 +267,7 @@
"pages": [
"setup-and-administration/managing-your-organization",
"setup-and-administration/github-app-permissions",
"setup-and-administration/trunk-sudo-app",
"setup-and-administration/support",
"setup-and-administration/billing",
"setup-and-administration/security"
Expand Down
78 changes: 78 additions & 0 deletions flaky-tests/autofix-ci-failures.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
---
title: "Autofix CI failures"
description: "Use Trunk and AI agents to automatically fix CI failures"
---

Trunk can return targeted information about CI failures, enabling AI agents and automation tools to analyze and fix issues automatically.

### Prerequisites

To use the Autofix CI Failures feature, you'll need to have:

* Your repository set up to [upload test results to Trunk](/flaky-tests/get-started)

### Cursor CI Autofix

You can set up a [Cursor Automation](https://cursor.com/automations) to automatically fix CI failures by connecting to Trunk's CI failure investigation data via MCP. This is an extension of the Cursor `CI Autofix` template.

Set up the Trunk MCP using [Bearer Authentication](/flaky-tests/use-mcp-server/configuration/bearer-auth).

<Accordion title="Recommended Cursor Automation Prompt">

```json
{
"name": "CI Autofix v1",
"description": "Detect CI failures on main and automatically open PRs",
"triggers": [
{
"git": {
"ciCompleted": {
"repos": [
"https://github.com/<repo>"
],
"condition": 1,
"ignoreBaseFailures": true
}
}
}
],
"actions": [
{
"gitPr": {}
},
{
"mcp": {
"server": {
"name": "trunk"
}
}
}
],
"prompts": [
{
"prompt": "Your task is to fix CI failures on PRs..."
}
],
"memoryEnabled": true,
"scope": "team_editable_user",
"templateId": "ci-autofix"
}
```

</Accordion>

We recommend the following conventions:

* Version your Automation names for more clarity (e.g., "CI Autofix v1")
* Refine the prompt to avoid scanning GitHub logs in order to save time and tokens
* Be specific about your repository's conventions and common failure patterns

<Info>
Currently Cursor will create a pull request with a base of `main`. You will need to adjust the pull request base if you want to merge the fix into your PR.
</Info>

### Claude Code Routines

<Info>
**Coming soon.** Set up Claude Routines to autofix CI failures
</Info>
73 changes: 73 additions & 0 deletions flaky-tests/autofix-flaky-tests.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
title: "Autofix flaky tests"
description: "Automatically investigate flaky tests and raise fix pull requests"
---

Trunk can automatically investigate flaky tests in your codebase and raise fix pull requests with suggested solutions.

### Prerequisites

To use the Autofix Flaky Tests feature, you'll need:

1. Beta access via waitlist (reach out to us on [Slack](https://slack.trunk.io))
2. The "Investigate Flaky Tests" setting enabled in your workspace
3. Active installation of the [Trunk GitHub App](/setup-and-administration/github-app-permissions)

### Auto-Investigate Flaky Tests

Once enabled, any time that Trunk [detects a flaky test](/flaky-tests/detection), Trunk analyzes the failure patterns, failure output, and git history of the test to provide a number of insights.

Flaky tests can also be analyzed manually via the UI and via the [MCP server](/flaky-tests/use-mcp-server/mcp-tool-reference/fix-flaky-test).

### Autofix with Cursor Automations

Whenever an investigation is completed, Trunk will emit a [webhook](/flaky-tests/webhooks) for `test_case.investigation_completed`. Enable webhooks via [Svix](/flaky-tests/webhooks).

You can then set up a [Cursor Automation](https://cursor.com/automations) to trigger when webhooks are received.

<Accordion title="Recommended Cursor Automation Config">

```json
{
"name": "Autofix Flaky Tests v1",
"triggers": [
{
"webhook": {}
}
],
"actions": [],
"prompts": [
{
"prompt": "Your task is to fix flaky tests in this repo using provided insights..."
}
],
"memoryEnabled": true,
"scope": "private",
"gitConfig": {
"repo": "https://github.com/<repo>",
"repos": [
"https://github.com/<repo>"
],
"branch": "main"
}
}
```

</Accordion>

We recommend the following conventions:

* Version your Automation names for more clarity.
* Configure the Svix endpoint with the Cursor Bearer token.
* Webhooks are configured for your entire organization, so you will need to use [Svix transformations](https://docs.svix.com/transformations) or filter out events that are not for your intended repository.
* Be specific about conventions and antipatterns for your repository. You will need to refine the Automation prompt to suit your needs.
* If your CI setup allows it, prompt Cursor to run the tests to verify them.

### What's next?

* Continue to monitor your tests to confirm the flaky test fixes are effective
* Investigations can be triggered and applied via [MCP](/flaky-tests/use-mcp-server/mcp-tool-reference/fix-flaky-test)

<Info>
**Coming soon.** Set up Claude Routines to autofix flaky tests
</Info>
27 changes: 15 additions & 12 deletions flaky-tests/detection.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Flake Detection automatically identifies problematic tests in your test suite by

Each monitor independently observes your test runs and tracks two states per test: **active** (problematic behavior detected) or **inactive** (no problematic behavior). A test's overall status is determined by combining all of its monitors, with the most severe status winning:

| Priority | Status | Condition |
| -------- | ----------- | --------------------------------------------------------------------- |
| Highest | **Broken** | Any enabled broken-type threshold monitor is active for this test |
| Middle | **Flaky** | Any enabled flaky-type monitor (threshold or pass-on-retry) is active |
| Lowest | **Healthy** | No active monitors |
| Priority | Status | Condition |
| -------- | ----------- | ------------------------------------------------------------------------------------------ |
| Highest | **Broken** | Any enabled broken-type monitor (failure rate or failure count) is active for this test |
| Middle | **Flaky** | Any enabled flaky-type monitor (failure rate, failure count, or pass-on-retry) is active |
| Lowest | **Healthy** | No active monitors |

If a test triggers both a broken monitor and a flaky monitor simultaneously, it shows as **Broken**. When the broken monitor resolves (e.g., you fix the regression and the failure rate drops), the test transitions to **Flaky** if a flaky monitor is still active, or to **Healthy** if no monitors remain active.

Expand All @@ -22,24 +22,27 @@ A test stays in its detected state until every relevant monitor that flagged it

When you disable or delete a monitor, it is immediately set to **resolved** for every test case in the repo. This triggers a status re-evaluation for all affected tests. If the disabled monitor was the only active monitor for a test, that test transitions to healthy. If other monitors are still active, the test remains in the most severe active state.

For example, if you have a broken threshold monitor and a flaky pass-on-retry monitor, and you disable the broken monitor, any test that was only flagged by the broken monitor will become healthy. A test flagged by both will transition from broken to flaky (because pass-on-retry is still active).
For example, if you have a broken failure rate monitor and a flaky pass-on-retry monitor, and you disable the broken monitor, any test that was only flagged by the broken monitor will become healthy. A test flagged by both will transition from broken to flaky (because pass-on-retry is still active).

## Monitor Types

| Monitor | What it detects | Detection type | Plan availability | Default state |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | --------------- | ----------------- | ------------- |
| [**Pass-on-Retry**](/flaky-tests/detection/pass-on-retry-monitor) | A test fails then passes on the same commit (retry after failure) | Flaky | Team and above | Enabled |
| [**Threshold**](/flaky-tests/detection/threshold-monitor) | Failure rate exceeds a configured percentage over a time window | Flaky or Broken | Paid plans | Disabled |
| Monitor | What it detects | Detection type | Plan availability | Default state |
| -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | --------------- | ----------------- | ------------- |
| [**Pass-on-Retry**](/flaky-tests/detection/pass-on-retry-monitor) | A test fails then passes on the same commit (retry after failure) | Flaky | Team and above | Enabled |
| [**Failure Rate**](/flaky-tests/detection/failure-rate-monitor) | Failure rate exceeds a configured percentage over a time window | Flaky or Broken | Paid plans | Disabled |
| [**Failure Count**](/flaky-tests/detection/failure-count-monitor) | A test accumulates a configured number of failures in a rolling window | Flaky or Broken | Paid plans | Disabled |

You can run multiple monitors simultaneously. For example, you might use pass-on-retry to catch classic retry-based flakiness while also running threshold monitors scoped to different branches. A common pattern is to pair a broken-type threshold monitor (catching consistently failing tests) with a flaky-type threshold monitor (catching intermittently failing tests). See [Threshold Monitor: Recommended Configurations](/flaky-tests/detection/threshold-monitor#recommended-configurations) for details.
You can run multiple monitors simultaneously. For example, you might use pass-on-retry to catch classic retry-based flakiness while also running failure rate monitors scoped to different branches. A common pattern is to pair a broken-type failure rate monitor (catching consistently failing tests) with a flaky-type failure rate monitor (catching intermittently failing tests). See [Failure Rate Monitor: Recommended Configurations](/flaky-tests/detection/failure-rate-monitor#recommended-configurations) for details.

The [failure count monitor](/flaky-tests/detection/failure-count-monitor) complements failure rate monitors by reacting to individual failures rather than failure rates. Use it on branches where any failure is a meaningful signal, like `main` or merge queue branches.

If you need to manually flag a test that automated monitors haven't caught, use [Flag as Flaky](/flaky-tests/detection/flag-as-flaky) from the test detail page.

## Branch-Aware Detection

Tests often behave differently depending on where they run. Failures on `main` are usually unexpected and signal flakiness. Failures on PR branches may be expected during active development. Merge queue failures are suspicious because the code has already passed PR checks.

Rather than applying a single set of branch rules automatically, Trunk gives you control over how detection treats different branches through **branch scoping** on threshold monitors. You can create separate monitors with different thresholds and windows for your stable branch, PR branches, and merge queue branches. See [Threshold Monitor: Recommended configurations](/flaky-tests/detection/threshold-monitor#recommended-configurations) for specific guidance.
Rather than applying a single set of branch rules automatically, Trunk gives you control over how detection treats different branches through **branch scoping** on failure rate monitors. You can create separate monitors with different thresholds and windows for your stable branch, PR branches, and merge queue branches. See [Failure Rate Monitor: Recommended configurations](/flaky-tests/detection/failure-rate-monitor#recommended-configurations) for specific guidance.

Pass-on-retry detection is branch-agnostic. It flags any test that fails and passes on the same commit, regardless of which branch the test ran on.

Expand Down
Loading