1. Find the tests whose status changed
Every test has astatus (healthy, flaky or broken) and a statusSince, the time that status began. A test changed this week if its statusSince is within the last seven days. Don’t look for those tests by listing every test. An organization can have hundreds of thousands of healthy tests. Make two short lists instead, paging through each with nextCursor:
- Turned flaky or broken. Call
GET /v2/testswithstatus=flaky,broken,include=metricsandperiod=7d. The metrics give each test’s failure rate for the week. - Turned healthy. A test recovers when a monitor stops flagging it, and the monitor records that as an event. Call
GET /v2/detection-eventswithisClassificationEvent=true,monitorStatus=inactiveanddetectedAfterset to the start of the week. For each test in those events that isn’t in the first list, callGET /v2/tests/{testId}withinclude=metrics&period=7d.
statusSince is within the week. Add testCollectionId to the list calls to send a digest for one collection.
Use
statusSince rather than newlyFlaky=true. newlyFlaky returns only tests that became flaky for the first time. It leaves out a test that was fixed and then turned flaky again, and a test that turned broken. Detection events don’t include a test someone flagged as flaky by hand. So if someone clears that flag and the test turns healthy, the test isn’t in the digest.2. Compare with the start of the week
A test can change status more than once in a week. For example, it can turn flaky on Monday and recover on Thursday. For each test from step 1, callGET /v2/tests/{testId}/status-history with startedAfter set to the start of the week. Transitions are listed newest first, so the last one’s fromStatus was the test’s status when the week began.
- If that status is the same as the test’s current
status, the test changed and then changed back. Leave it out. - Otherwise, report the test as
fromStatus→status, such ashealthy → flakyorbroken → healthy.
activeMonitors lists the monitors that flagged the test, if you want to show why it changed.
3. Group by owner and post to Slack
Each test hascodeowners, such as @acme/payments. Group the tests by owner. A test with more than one owner goes under each of them, and a test with none goes under Unowned. Then post the groups to a Slack incoming webhook. Each test’s url links to its page in Trunk.
This script does all three steps. It needs Node.js 18 or later and an API key. Reading tests needs no permissions, so leave them all off.
flaky-digest.mjs
byOwner and post it to the team’s webhook URL.
4. Run it every week
Run the script from any scheduler, such as cron or a scheduled CI job. This GitHub Actions workflow posts every Monday at 14:00 UTC. Store the API key and the Slack webhook URL as repository secrets:.github/workflows/flaky-digest.yml
Related
- Fix Flaky Tests. Hand the tests in your digest to a coding agent.
- Alert When a Test Escalates. Get an alert as soon as a test gets worse, instead of once a week.
- Managing Detected Flaky Tests. Build a triage process around the digest.