1. Configure a timeout inflation monitor
The timeout inflation monitor flags a test when its failures take far longer than its passes, which means its failures are waiting out a timeout. Create one in the test collection’s monitor settings and use the preview panel to see which tests it would flag before you save. Give it a label such astimeout-inflated, so the tests it flags are easy to list.
These values suit this recipe. Adjust each one by what the right-hand column says it depends on.
2. Find the tests to tighten
Start the agent in one of two ways:- From a webhook. Subscribe the agent’s endpoint to the
test_case.monitor_status_changedevent and act whenmonitor.typeistimeout_inflationandmonitor.statusisactive. The event’sevidenceholds the passing and failing durations and their ratio. - On a schedule. List the tests the monitor flags with
GET /v2/monitors/{monitorId}/tests, or withGET /v2/testsandlabel=timeout-inflated.
GET /v2/tests with sort=durationP95, period=7d and include=metrics, and have the agent compare each test’s runtime with its configured timeout.
3. Gather context
The P95 duration on the test covers failing runs too, so a test that keeps waiting out its timeout pushes its own P95 up. Have the agent use passing runs only.
- Jest and Vitest: the timeout argument to
itortest,jest.setTimeout,vi.setConfig, andtestTimeoutin the config. - Playwright:
test.setTimeout,test.slow(), andtimeoutandexpect.timeoutinplaywright.config. The timeout applies to each retry. - pytest:
@pytest.mark.timeoutandtimeoutin the pytest config. - Go:
go test -timeoutandcontext.WithTimeoutinside tests. - JUnit:
@Timeout. - CI jobs:
timeout-minutesand its equivalents.
4. Propose tighter values or find the cause
Lower the timeout when the test’s passing durations are steady. Set it to twice the passing P95, at least 1.25 times the slowest pass in the window, and no lower than the framework’s default for that kind of test. Round up to a round number. Propose a change only when the current timeout is at least three times the new value.- Use the slowest CI job: if the test is slow only on one runner type, the timeout must fit that one.
- Change a shared timeout only when every test it covers fits the new value. Otherwise, give the test a timeout of its own.
5. Track
- Link the pull request’s ticket with
POST /v2/tests/{testId}/ticket, or create one withPOST /v2/tests/{testId}/create-ticket. - After the pull request merges, the monitor resolves the test once its failures fall under the resolution ratio, and removes the label.
- If a test starts timing out on healthy runs, the new timeout is too tight. Raise it to fit the slowest pass, or find the cause of the slowness as in step 4.
Related
- Timeout inflation monitor. How the monitor compares failing and passing durations.
- Fix Flaky Tests. Have the agent fix a test whose failures aren’t timeouts.