> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trunk.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Explain a Failed Pull Request

> Comment on a red pull request with which failed tests are known to be flaky and which are probably real.

When a pull request goes red, its author has to work out which failures they caused. A failed test might be quarantined, already known to be flaky, already failing on your main branch, or new. Trunk knows which, so a step at the end of your test job can tell the author in a pull request comment:

> ❌ **4 tests failed. 1 probably needs your attention.**
> 🔴 `CheckoutTotal › applies discount`: passes on main and hasn't failed on other pull requests. **Probably caused by this change.**
> 🟡 `Cart › syncs across tabs`: known flaky since Nov 2, failing 18% of runs (FLAKY-123). Retry it.
> 🟠 `Auth › refresh token`: it also failed 3 times on your main branch this week. Not caused by this change.
> 🟢 `Search › debounces`: quarantined, so it doesn't fail the job.

The step reads the list of failed tests from the summary that the uploader writes, looks each test up with the [API](https://docs.trunk.io/api.md), and posts the comment.

## 1. Write the run's summary

Upload your results with `--summary-output-file`. The uploader then writes a JSON summary of the run, including every failed test:

```json expandable theme={"system"}
{
  "schemaVersion": 2,
  "type": "UPLOAD",
  "quarantining": "CHECKED",
  "failures": [
    {
      "key": { "type": "FILE", "name": "src/checkout.test.ts" },
      "tests": [
        {
          "name": "CheckoutTotal › applies discount",
          "file": "src/checkout.test.ts",
          "id": "8f14e45f-ceea-467a-9575-7a2b5c1d3e90",
          "quarantined": false,
          "link": "https://app.trunk.io/acme/flaky-tests/collections/aB3xY9kQ/tests/8f14e45f-ceea-467a-9575-7a2b5c1d3e90"
        }
      ]
    }
  ],
  "quarantined": [ ... ]
}
```

* `failures` holds the tests that failed the run. `quarantined` holds the ones that failed but were [quarantined](../quarantining/), so they didn't fail the run.
* Each test's `id` is its Trunk test ID, the `testId` the API takes. It is `null` if Trunk couldn't identify the test collection for the run, for example when the uploader couldn't reach Trunk.
* `quarantining` says whether Trunk was asked which tests are quarantined. When it is `LOOKUP_FAILED`, no test is marked quarantined.

`trunk-analytics-cli summary-output-schema` prints the summary's full JSON Schema.

In GitHub Actions, the [Analytics Uploader Action](https://github.com/trunk-io/analytics-uploader) writes the summary for you and gives its path as the `summary-output-file` output. With the CLI, pass `--summary-output-file <path>`, or set `TRUNK_SUMMARY_OUTPUT_FILE`.

## 2. Look up each failed test

For each test with an `id`, call [`GET /v2/tests/{testId}`](https://docs.trunk.io/api/tests/get-a-test) with `include=metrics&period=7d`. One call returns everything the comment needs:

* `status` (`healthy`, `flaky` or `broken`) and `statusSince`, when that status began.
* `metrics.failureRate`, the share of the test's runs that failed this week.
* `ticket`, the ticket linked to the test, if there is one.
* `url`, the test's page in Trunk.

## 3. Check where else the test fails

A test that Trunk sees as healthy still might not be the author's fault. Two calls to [`GET /v2/tests/{testId}/test-runs`](https://docs.trunk.io/api/tests/list-a-tests-runs), each with `conclusion=failure` and `startedAfter` set to a week ago, tell you where else it has failed:

* **On your main branch.** Pass `branches=main`. If you use [Trunk Merge Queue](https://docs.trunk.io/merge-queue/merge-queue.md), add its branches, `branches=main,trunk-merge/*`, because some test suites run in the queue more often than on `main`. A failure there means the test was failing before this pull request.
* **On other pull requests.** Pass `pullRequestOnly=true` and count the distinct `pullRequest.number` values, leaving out this pull request. A test that failed on several unrelated pull requests may be flaky but not yet detected.

## 4. Decide what to tell the author

Check each test against these rules in order and use the first one that matches:

| Trunk says | Comment |
| - | - |
| The summary marks it `quarantined` | 🟢 Quarantined. It didn't fail the job. |
| `status` is `flaky` | 🟡 Known flaky. Retry it. |
| `status` is `broken`, or it failed on your main branch this week | 🟠 Not caused by this change. |
| It failed on 3 or more other pull requests this week | 🟡 Possibly flaky. Retry it once. |
| None of the above | 🔴 Probably caused by this change. |
| The summary has no `id` | ⚪ Trunk couldn't look it up. Link to it only. |

A quarantined test with a failure rate near 100% isn't flaky: it fails every time. That's worth saying in the comment so someone fixes it, even though it doesn't fail the job.

## 5. Post the comment

This script does steps 2 to 5 and creates or updates one comment on the pull request. It needs Node.js 18 or later and an [API key](https://docs.trunk.io/api/authentication#api-keys). Reading tests needs no permissions, so leave them all off.

```javascript explain-failed-tests.mjs expandable theme={"system"}
import { readFileSync } from "node:fs";

const API = "https://api.trunk.io/v2";
const headers = { Authorization: `Bearer ${process.env.TRUNK_API_KEY}` };
const { SUMMARY_FILE, BASELINE_BRANCHES = "main", GITHUB_REPOSITORY, PR_NUMBER, GITHUB_TOKEN } = process.env;
const MARKER = `<!-- trunk-explain-failed-tests:${process.env.GITHUB_JOB ?? ""} -->`;
const weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString();

async function get(path, params) {
  const url = new URL(API + path);
  for (const [key, value] of Object.entries(params)) url.searchParams.set(key, value);
  const res = await fetch(url, { headers });
  if (!res.ok) throw new Error(`${res.status} ${path}: ${await res.text()}`);
  return res.json();
}

async function inBatches(items, size, fn) {
  const results = [];
  for (let i = 0; i < items.length; i += size) {
    results.push(...(await Promise.all(items.slice(i, i + size).map(fn))));
  }
  return results;
}

const percent = (rate) => `${Math.round(rate * 100)}%`;
const runs = (testId, params) =>
  get(`/tests/${testId}/test-runs`, { conclusion: "failure", startedAfter: weekAgo, limit: 100, ...params });

async function explain(failed) {
  const name = `[\`${failed.name}\`](${failed.link})`;
  const quarantined = `🟢 ${name}: quarantined, so it didn't fail the job.`;
  const fallback = () =>
    failed.quarantined
      ? { order: 5, line: quarantined }
      : { order: 1, line: `⚪ ${name}: Trunk couldn't look this test up.` };
  if (!failed.id) return fallback();

  try {
    const test = await get(`/tests/${failed.id}`, { include: "metrics", period: "7d" });
    const rate = test.metrics?.failureRate ?? 0;
    const ticket = test.ticket ? ` (${test.ticket.key})` : "";

    if (failed.quarantined) {
      const note = rate >= 0.95 ? ` It failed ${percent(rate)} of runs this week, so it needs fixing.` : "";
      return { order: 5, line: quarantined + note };
    }
    if (test.status === "flaky") {
      const since = test.statusSince ? ` since ${test.statusSince.slice(0, 10)}` : "";
      return { order: 3, line: `🟡 ${name}: known flaky${since}, failing ${percent(rate)} of runs${ticket}. Retry it.` };
    }

    const [onMain, onPrs] = await Promise.all([
      runs(failed.id, { branches: BASELINE_BRANCHES }),
      runs(failed.id, { pullRequestOnly: true }),
    ]);
    if (test.status === "broken" || onMain.data.length > 0) {
      const where = test.status === "broken" ? "Trunk classifies it as broken" : `it also failed ${onMain.data.length} times on your main branch this week`;
      return { order: 4, line: `🟠 ${name}: ${where}${ticket}. Not caused by this change.` };
    }
    const otherPrs = new Set(onPrs.data.map((run) => run.pullRequest?.number).filter((n) => n && n !== Number(PR_NUMBER))).size;
    if (otherPrs >= 3) {
      return { order: 2, line: `🟡 ${name}: failed on ${otherPrs} other pull requests this week. It may be flaky. Retry it once.` };
    }
    return { order: 0, line: `🔴 ${name}: passes on your main branch and hasn't failed on other pull requests. **Probably caused by this change.**` };
  } catch (error) {
    console.error(error);
    return fallback();
  }
}

const summary = JSON.parse(readFileSync(SUMMARY_FILE, "utf8"));
const failed = [
  ...(summary.failures ?? []).flatMap((group) => group.tests),
  ...(summary.quarantined ?? []).flatMap((group) => group.tests),
];
if (failed.length === 0) process.exit(0);

const rows = (await inBatches(failed, 4, explain)).sort((a, b) => a.order - b.order);
const real = rows.filter((row) => row.order === 0).length;
const body = [
  MARKER,
  `❌ **${failed.length} tests failed. ${real} probably ${real === 1 ? "needs" : "need"} your attention.**`,
  "",
  ...rows.map((row) => `- ${row.line}`),
].join("\n");

const github = (path, init = {}) =>
  fetch(`https://api.github.com/repos/${GITHUB_REPOSITORY}${path}`, {
    ...init,
    headers: { Authorization: `Bearer ${GITHUB_TOKEN}`, Accept: "application/vnd.github+json" },
  });
const comments = await (await github(`/issues/${PR_NUMBER}/comments?per_page=100`)).json();
const existing = comments.find((comment) => comment.body?.startsWith(MARKER));
const res = existing
  ? await github(`/issues/comments/${existing.id}`, { method: "PATCH", body: JSON.stringify({ body }) })
  : await github(`/issues/${PR_NUMBER}/comments`, { method: "POST", body: JSON.stringify({ body }) });
if (!res.ok) throw new Error(`${res.status} posting the comment: ${await res.text()}`);
```

The script looks up four tests at a time. Each lookup makes at most three requests, so a run with hundreds of failures stays well inside the API's [rate limits](https://docs.trunk.io/api/rate-limits.md). A lookup that fails is listed as one Trunk couldn't look up, rather than failing the step.

Run it in your test job, after the upload. In GitHub Actions:

```yaml expandable theme={"system"}
jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write # to comment on the pull request
    steps:
      - uses: actions/checkout@v4

      - name: Run Tests
        run: ...

      - name: Upload Test Results to Trunk.io
        id: trunk
        if: ${{ !cancelled() }}
        continue-on-error: true
        uses: trunk-io/analytics-uploader@v2
        with:
          junit-paths: "**/junit.xml"
          org-slug: <TRUNK_ORG_URL_SLUG>
          test-collection-id: <TRUNK_TEST_COLLECTION_ID>
          token: ${{ secrets.TRUNK_API_TOKEN }}

      - name: Explain failed tests
        if: ${{ !cancelled() && github.event_name == 'pull_request' && steps.trunk.outputs.summary-output-file != '' }}
        run: node .github/scripts/explain-failed-tests.mjs
        env:
          SUMMARY_FILE: ${{ steps.trunk.outputs.summary-output-file }}
          BASELINE_BRANCHES: main
          TRUNK_API_KEY: ${{ secrets.TRUNK_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          GITHUB_TOKEN: ${{ github.token }}
```

Each job gets its own comment, so a pull request with several test jobs gets one comment per job that had failures. A rerun of the same job updates its comment rather than adding another.

> [!NOTE]
>
> `TRUNK_API_TOKEN` is the organization token the uploader uses. `TRUNK_API_KEY` is an [API key](https://docs.trunk.io/api/authentication#api-keys) for the API. They are different credentials.

## Related

* [Quarantining](../quarantining/). How quarantined failures stop failing your CI.
* [Flake Detection](../detection/). How Trunk decides a test is flaky or broken.
* [CLI reference](https://docs.trunk.io/flaky-tests/reference/cli-reference.md). The uploader's options, including `--summary-output-file`.
* [Alert When a Test Escalates](https://docs.trunk.io/flaky-tests/recipes/alert-on-test-escalation.md). Hear about a test when its status changes, rather than when it fails.
