Skip to main content
When a pull request goes red, its author has to work out which failures they caused. A failed test might be quarantined, already known to be flaky, already failing on your main branch, or new. Trunk knows which, so a step at the end of your test job can tell the author in a pull request comment:
❌ 4 tests failed. 1 probably needs your attention. 🔴 CheckoutTotal › applies discount: passes on main and hasn’t failed on other pull requests. Probably caused by this change. 🟡 Cart › syncs across tabs: known flaky since Nov 2, failing 18% of runs (FLAKY-123). Retry it. 🟠 Auth › refresh token: it also failed 3 times on your main branch this week. Not caused by this change. 🟢 Search › debounces: quarantined, so it doesn’t fail the job.
The step reads the list of failed tests from the summary that the uploader writes, looks each test up with the API, and posts the comment.

1. Write the run’s summary

Upload your results with --summary-output-file. The uploader then writes a JSON summary of the run, including every failed test:
  • failures holds the tests that failed the run. quarantined holds the ones that failed but were quarantined, so they didn’t fail the run.
  • Each test’s id is its Trunk test ID, the testId the API takes. It is null if Trunk couldn’t identify the test collection for the run, for example when the uploader couldn’t reach Trunk.
  • quarantining says whether Trunk was asked which tests are quarantined. When it is LOOKUP_FAILED, no test is marked quarantined.
trunk-analytics-cli summary-output-schema prints the summary’s full JSON Schema. In GitHub Actions, the Analytics Uploader Action writes the summary for you and gives its path as the summary-output-file output. With the CLI, pass --summary-output-file <path>, or set TRUNK_SUMMARY_OUTPUT_FILE.

2. Look up each failed test

For each test with an id, call GET /v2/tests/{testId} with include=metrics&period=7d. One call returns everything the comment needs:
  • status (healthy, flaky or broken) and statusSince, when that status began.
  • metrics.failureRate, the share of the test’s runs that failed this week.
  • ticket, the ticket linked to the test, if there is one.
  • url, the test’s page in Trunk.

3. Check where else the test fails

A test that Trunk sees as healthy still might not be the author’s fault. Two calls to GET /v2/tests/{testId}/test-runs, each with conclusion=failure and startedAfter set to a week ago, tell you where else it has failed:
  • On your main branch. Pass branches=main. If you use Trunk Merge Queue, add its branches, branches=main,trunk-merge/*, because some test suites run in the queue more often than on main. A failure there means the test was failing before this pull request.
  • On other pull requests. Pass pullRequestOnly=true and count the distinct pullRequest.number values, leaving out this pull request. A test that failed on several unrelated pull requests may be flaky but not yet detected.

4. Decide what to tell the author

Check each test against these rules in order and use the first one that matches: A quarantined test with a failure rate near 100% isn’t flaky: it fails every time. That’s worth saying in the comment so someone fixes it, even though it doesn’t fail the job.

5. Post the comment

This script does steps 2 to 5 and creates or updates one comment on the pull request. It needs Node.js 18 or later and an API key. Reading tests needs no permissions, so leave them all off.
explain-failed-tests.mjs
The script looks up four tests at a time. Each lookup makes at most three requests, so a run with hundreds of failures stays well inside the API’s rate limits. A lookup that fails is listed as one Trunk couldn’t look up, rather than failing the step. Run it in your test job, after the upload. In GitHub Actions:
Each job gets its own comment, so a pull request with several test jobs gets one comment per job that had failures. A rerun of the same job updates its comment rather than adding another.
TRUNK_API_TOKEN is the organization token the uploader uses. TRUNK_API_KEY is an API key for the API. They are different credentials.