❌ 4 tests failed. 1 probably needs your attention. 🔴The step reads the list of failed tests from the summary that the uploader writes, looks each test up with the API, and posts the comment.CheckoutTotal › applies discount: passes on main and hasn’t failed on other pull requests. Probably caused by this change. 🟡Cart › syncs across tabs: known flaky since Nov 2, failing 18% of runs (FLAKY-123). Retry it. 🟠Auth › refresh token: it also failed 3 times on your main branch this week. Not caused by this change. 🟢Search › debounces: quarantined, so it doesn’t fail the job.
1. Write the run’s summary
Upload your results with--summary-output-file. The uploader then writes a JSON summary of the run, including every failed test:
failuresholds the tests that failed the run.quarantinedholds the ones that failed but were quarantined, so they didn’t fail the run.- Each test’s
idis its Trunk test ID, thetestIdthe API takes. It isnullif Trunk couldn’t identify the test collection for the run, for example when the uploader couldn’t reach Trunk. quarantiningsays whether Trunk was asked which tests are quarantined. When it isLOOKUP_FAILED, no test is marked quarantined.
trunk-analytics-cli summary-output-schema prints the summary’s full JSON Schema.
In GitHub Actions, the Analytics Uploader Action writes the summary for you and gives its path as the summary-output-file output. With the CLI, pass --summary-output-file <path>, or set TRUNK_SUMMARY_OUTPUT_FILE.
2. Look up each failed test
For each test with anid, call GET /v2/tests/{testId} with include=metrics&period=7d. One call returns everything the comment needs:
status(healthy,flakyorbroken) andstatusSince, when that status began.metrics.failureRate, the share of the test’s runs that failed this week.ticket, the ticket linked to the test, if there is one.url, the test’s page in Trunk.
3. Check where else the test fails
A test that Trunk sees as healthy still might not be the author’s fault. Two calls toGET /v2/tests/{testId}/test-runs, each with conclusion=failure and startedAfter set to a week ago, tell you where else it has failed:
- On your main branch. Pass
branches=main. If you use Trunk Merge Queue, add its branches,branches=main,trunk-merge/*, because some test suites run in the queue more often than onmain. A failure there means the test was failing before this pull request. - On other pull requests. Pass
pullRequestOnly=trueand count the distinctpullRequest.numbervalues, leaving out this pull request. A test that failed on several unrelated pull requests may be flaky but not yet detected.
4. Decide what to tell the author
Check each test against these rules in order and use the first one that matches:
A quarantined test with a failure rate near 100% isn’t flaky: it fails every time. That’s worth saying in the comment so someone fixes it, even though it doesn’t fail the job.
5. Post the comment
This script does steps 2 to 5 and creates or updates one comment on the pull request. It needs Node.js 18 or later and an API key. Reading tests needs no permissions, so leave them all off.explain-failed-tests.mjs
TRUNK_API_TOKEN is the organization token the uploader uses. TRUNK_API_KEY is an API key for the API. They are different credentials.Related
- Quarantining. How quarantined failures stop failing your CI.
- Flake Detection. How Trunk decides a test is flaky or broken.
- CLI reference. The uploader’s options, including
--summary-output-file. - Alert When a Test Escalates. Hear about a test when its status changes, rather than when it fails.