1. Find the tests that failed
Choose the approach that fits how you test releases.From a pre-release workflow
If a workflow tests the release before you cut it, have the analytics CLI write a summary of the upload, and give that file to the agent.- In the workflow, upload the results with analytics CLI 1.0.0 or newer, adding
--summary-output-file trunk-summary.json. - Pass
trunk-summary.jsonto the agent, for example as a workflow artifact.
failuresholds the failed tests that weren’t quarantined, andquarantinedthe ones that were. Both are grouped by file.- Each test has its
name,file,failureMessage,failureTextandlink. - The last segment of
link(…/tests/<testId>) is the test’s ID in the API. Use it in the calls below, not the test’sidfield, which is the CLI’s own ID.
The summary reports each test’s final result in the report. A test that your runner retried and then passed counts as passed.
From a failure count monitor on release branches
Have Trunk label every test that fails on a release branch, then start from that label.- In the collection that holds the release branch’s results, create a failure count monitor:
- Branches: the release branch, or a pattern such as
release/*. - Failure count:
1. - Resolution timeout: long enough to cover your release cycle, so a test stays labeled after a single failure. The web app allows up to 14 days.
- Action: Apply labels, with a label such as
failed-on-release. See Automatic labeling from monitors.
- Branches: the release branch, or a pattern such as
- Have the agent list the labeled tests with
GET /v2/tests, usingtestCollectionIdandlabel=failed-on-release.
2. Enrich each test
Read quarantine from the test’s
quarantineState, not the run’s. A run’s quarantineState records what its upload reported.3. Report
Sort each test by its current state. Adjust these to your release policy:- Post a report grouped by severity. For each test, give its name,
codeowners, failure rate and a link to its latest failing run. - Track the tests that need attention with
POST /v2/tests/{testId}/add-labels, for examplerelease-risk, andPOST /v2/tests/{testId}/create-ticket. To fix them, see Fix Flaky Tests. - To hold the release on the result, have the agent exit non-zero when any test needs attention, before your tag or publish job.
- Don’t let the agent quarantine tests or mark them flaky to improve the result.
Related
- Fix Flaky Tests. Hand the tests that need attention to a coding agent.
- Explain a Failed Pull Request. Use the same upload summary to explain failures on a pull request.
- Failure count monitor. How the monitor labels tests that fail on a branch.