Quarantining

Quarantining lets you isolate failures for known flaky tests so they don't fail your CI jobs, while still continuing to run them. Quarantining flaky tests lets you mitigate the negative effects of flaky tests without disabling any tests. Quarantined failures will still be uploaded to Trunk Flaky Tests and displayed in PRs through Universal Test Reports, allowing you to easily triage failures and identify real issues surfaced by flaky tests.

Prerequisites of Quarantining

Before a test can be quarantined, the test must be marked as flaky by Trunk Flaky Tests. You can tell if a test is flaky by the ⚠️ label or by checking its Status History.

A test's status history shows it labeled as flaky.

Quarantining Modes

Quarantining operates in one of three modes, disabled, enabled, and preview. If you’re trying quarantining for the first time in a project, you can enable the preview mode, which will show you what could be quarantined but will not affect your CI job results.

Mode	Behavior
Disabled	No tests will be quarantined.
Enabled	If CI jobs use the Analytics Uploader with Quarantining enabled, failed flaky tests will be quarantined. If all failed tests are quarantined, the CI job result will be overridden as a pass.
Preview	If CI jobs use the Analytics Uploader with Quarantining enabled, failed flaky tests will be displayed as quarantined in the Flaky Test web app, but the CI jobs will not be overridden as a pass.

Enable Quarantining

You can enable quarantining by navigating to Settings > Repositories > Flaky Tests > Quarantine Flaky Tests and selecting enabled in the drop-down.

After enabling quarantining on Trunk Flaky Tests, future CI jobs with quarantining enabled on the Analytics Uploader will begin quarantining failed flaky tests.

Update CI Jobs

Before tests can be quarantined on a CI job, the test command needs to be run via the Analytics Uploader through the run: <COMMAND TO RUN TESTS> parameter and have quarantine: truepassed to the Analytics Uploader to enable quarantining.

With quarantining enabled, the Analytics Uploader will compare failed test cases against known flaky tests. If a test is known to be flaky, it will be quarantined. If all failed tests are flaky then the exit code of the test command will be overridden to return 0 which means the CI job will pass.

Here is an example of a GitHub workflow:

name: Upload Test Results to Trunk
on:
  workflow_dispatch:
jobs:
  upload-test-results:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Run tests and upload results
        continue-on-error: true
        quarantine: true
        run: <COMMAND TO RUN TESTS>
        env:
          token: ${{ secrets.TRUNK_API_TOKEN }}
          org-slug: my-trunk-org-slug
        with:
          junit-paths: <TEST OUTPUT PATH>

      - name: Upload results
        uses: trunk-io/analytics-uploader@main
        if: "!cancelled()"

Overriding Individual Tests

If you have tests that should never be quarantined or should always be quarantined regardless of their current health status, you can do this through overriding individual tests.

You can update the overriding settings on each test by navigating to the details page for a specific test and clicking the Quarantining (Repo Default) selector in the top left corner.

You can choose between one of three override settings:

Setting	Behavior
Repo Default	Quarantining behavior follows the repository settings. Learn more about quarantining modes
Always Quarantine	Quarantine a test failure even if the health status is healthy or broken.
Never Quarantine	Never quarantine failures, even if the health status is flaky and quarantining is enabled for the repo.

Audit Logs

Trunk provides audit logs for all setting changes and overwrites for individual tests. You can access the audit log by navigating to Settings > Repositories > Flaky Tests > Quarantine Flaky Tests > Audit logs.