Flaky Tests provides summary analytics about tests running on Pull Requests.
Last updated
Overview
Flaky Tests can post comments on GitHub pull requests to provide a summary of all the tests run on this PR, across multiple CI jobs or even CI systems, as well as whether those failures are due to flakiness, the history of the failures of that test, and more.
Each GitHub comment has a full summary report, containing all tests that passed, failed, flaked, skipped, and quarantined on this PR.
Each test case also contains the full stack trace attached when opened. You can also click on the Job run link which will take you to the full CI logs.
Configuration
If you have the Trunk GitHub App installed and are uploading JUnit XML test results on pull requests, expect to start seeing comments on your Pull Requests soon. If you prefer not to use the Trunk GitHub App you can still set up comments on your Pull Requests by providing Trunk with a GitHub access token.
Without Trunk GitHub App
It's recommended that the Trunk GitHub App be used to manage GitHub comments. If you need to generate comments without the Trunk GitHub app, you can do so with a service account and an API token.
Create a dedicated GitHub SVC account (Service Account) with access to the repositories in your GitHub Organization that Flaky Tests will comment on e.g., trunk-analytics-user.
On github.com, for trunk-analytics-user (or whichever user you wish to use), generate a Personal access token by navigating to Settings > Developer settings > Personal access token > Fine-grained tokens > Generate new token.
Name the new token something memorable. ex: trunk-flaky-tests-token.
The expiry time is up to you - however long you wish to try out Flaky Tests comments/how often you are willing to rotate the token. For a longer-term solution, consider installing the Trunk GitHub App.
Select the repositories you wish to enable comments on.
Permissions - you must enable Issues (Read and write) and Pull requests (Read and write). Note: It is expected that metadata permissions automatically change.
If everything looks good, scroll down to double check that your Overview for permissions looks something like the image below. If so, create the token.
Once the token is generated, go back to the Trunk App (app.trunk.io ) > click on your profile > Settings > Manage (under Organization) > Organization GitHub Token and enter the copied token into the text field, then finally press Submit.
You should see comments posted by your service account on your next PR.
GitHub Org Ownership
If you wish to set the resource owner to be a GitHub Organization, you should double check that this is allowed by navigating to your GitHub Organization > Settings > Personal access tokens > Settings. Make sure under "Fine-grained personal access tokens", you have "Allow access via fine-grained personal access tokens" selected.
Once the token is created, the Organization admin may need to approve the request for the token. This can be done by going to Github Organization > Settings > Personal access tokens > Pending requests. To confirm that the token was set, you should be able to see it under Active tokens.
Disable commenting
Pull Request comments are enabled by default. If you wish to disable the comments, you can do so by navigating to the Trunk App > click on your profile > Settings > RepoName. Scroll down to Test Analytics then toggle the Summary Flaky Tests Reports setting.
Troubleshooting
At any point, feel free to reach out to our team through Slack.
