Organizing tests with collections in Trunk Flaky Tests

A test collection is a named group of tests with its own settings. Each collection has independent flake detection, quarantining, and ticketing, so you can hold fast unit tests and fragile end-to-end tests to different standards. You create a collection in the Trunk app, then point a CI job at it by passing the collection’s ID to the uploader.

The Overview tab of a collection named trunk2-pr-e2e, showing its Collection ID, flaky and broken test counts, PRs impacted, three daily charts, and the top ten most unstable tests. — A collection's Overview tab, once test results are flowing.

Collections and repositories

Collections and repositories are many-to-many. A collection can include tests from multiple repositories, and a repository can be broken into multiple collections. This is decided by which collection you specify at upload time.

Three repositories uploading into two collections: the api repository sends its unit tests to one collection and its end-to-end tests to another, while the unit test collection also receives uploads from the web repository.

The Collection ID

Every collection has an ID: eight alphanumeric characters, unique across all of Trunk. It appears in the collection’s URL, on the collection’s page header, and in the ID column of the collections list, each with a button to copy it.

https://app.trunk.io/flaky-tests/collections/<collection-id>

The ID is what CI needs. Nothing else about a collection is used to route uploads.

Create a collection

Any member of your organization can create a collection. Enter a Collection name, and a description if you want one. Your first collection is created from the form Flaky Tests opens on; later ones from Create collection in the actions menu on the collections list. You land on the new collection. Its Uploads tab opens when an upload arrives; Tests waits until those results are ingested.

What a new collection starts with

A new collection starts with five monitors, so detection works as soon as test results arrive.

Monitor	Configuration	Action
Pass-on-retry	1 retry, 7 day resolve, every branch	Classify as flaky
Failure count	5 failures, 7 day resolve	Classify as flaky
Failure rate	90% over 24 hours, 5 sample minimum	Classify as broken
New test	7 days	Apply the New label
Skipped test	24 hour window, 7 day resolve	Apply the Skipped label

Every monitor but pass-on-retry watches main, master, develop, dev, and release/*. A collection has no repository until its first upload, so check those branch names against your own. Quarantining also starts enabled, with auto-quarantine off, so nothing is quarantined until you turn auto-quarantine on or set an override on a test. While your organization is migrating, it starts off instead, because your repositories’ settings are still governing uploads. You can change any of it from the collection’s Monitors and Settings tabs.

Upload test results to a collection

Pass the collection ID to the uploader alongside your organization slug. Both are required:

./trunk-analytics-cli upload \
  --junit-paths "test_output.xml" \
  --org-url-slug <TRUNK_ORG_URL_SLUG> \
  --test-collection-id <COLLECTION_ID> \
  --token $TRUNK_API_TOKEN

You can set the ID as the TRUNK_TEST_COLLECTION_ID environment variable instead, or pass it as the test-collection-id input to the uploader action. Everything else about your setup is unchanged. Adding the collection ID to a working upload step is the whole change. See Test Frameworks and CI Providers for the setup itself, and the CLI reference for the full flag list.

A CI job uploads to exactly one collection. To send results to more than one collection, add an upload step per collection.

Collection tabs

Tab	What it holds
Overview	Setup instructions before results arrive; the collection’s dashboard afterwards.
Tests	Every test in the collection, with health status, failure rates, and labels.
Uploads	Upload history for the collection.
Quarantining	Quarantine analytics for the collection.
Monitors	The collection’s flake detection monitors.
Settings	The collection’s name and description, quarantining, upload handling, and ticketing.

Finish setting up a collection

Each collection has a setup checklist, on its Overview tab before results arrive and on its Settings tab permanently. It tracks five steps:

Send your first upload
Ingest your test results
Review flake detection
Review quarantining
Review ticketing

The first two complete on their own once uploads are flowing. You complete the last three yourself, from the Monitors tab, Settings → Quarantining, and Settings → Ticketing. If you are not using ticketing, say so and that step is complete. Ticketing may already be done. Connections belong to your organization, so another collection may have set one up.

Permissions

Action	Admin	Member
View collections	Yes	Yes
Create a collection	Yes	Yes
Create, edit, delete, or mute a monitor	Yes	Yes
Connect, copy, or remove an organization ticketing integration	Yes	Yes
Choose a collection’s default ticketing connection	Yes	Yes
Create, link, or unlink a ticket	Yes	Yes
Edit a collection’s name or description	Yes	No
Change quarantining settings	Yes	No
Accept uploads from forked pull requests (set up)	Yes	No
Delete a collection	Yes	No

Delete a collection

Only organization admins can delete a collection.

Open the collection’s Settings tab.
Click Delete collection.
Confirm in the dialog.

The collection disappears from the collections list. Test results already uploaded to it are not deleted.

Questions

Ask us in Slack or email support@trunk.io.

​Collections and repositories

​The Collection ID

​Create a collection

​What a new collection starts with

​Upload test results to a collection

​Collection tabs

​Finish setting up a collection

​Permissions

​Delete a collection

​Questions

Collections and repositories

The Collection ID

Create a collection

What a new collection starts with

Upload test results to a collection

Collection tabs

Finish setting up a collection

Permissions

Delete a collection

Questions