Pull request comments
Flaky Tests provides summary analytics about tests running on Pull Requests
Flaky Tests can post comments on GitHub pull requests that summarize test results across CI jobs. These comments indicate which failures are flaky and include the test’s failure history and related context.
Note: Flaky Tests will only post a comment when there are failing tests.
Each GitHub comment includes a summary report showing all tests that passed, failed, flaked, were skipped, or were quarantined on the PR.
Each test case includes the full stack trace when expanded, and the job run link takes you to the complete CI logs.
Configuration
If you have the Trunk GitHub App installed and are uploading JUnit XML test results on pull requests, expect to start seeing comments on your Pull Requests soon. If you prefer not to use the Trunk GitHub App, you can still set up comments on your Pull Requests by providing Trunk with a GitHub access token.
Without Trunk GitHub App
It's recommended that the Trunk GitHub App be used to manage GitHub comments. If you need to generate comments without the Trunk GitHub app, you can do so with a service account and an API token.
Create a dedicated GitHub SVC account (Service Account) with access to the repositories in your GitHub Organization that Flaky Tests will comment on e.g.,
trunk-analytics-user.On github.com, for
trunk-analytics-user(or whichever user you wish to use), generate a Personal access token by navigating to Settings > Developer settings > Personal access token > Fine-grained tokens > Generate new token.Name the new token something memorable. ex:
trunk-flaky-tests-token.The expiry time is up to you - however long you wish to try out Flaky Tests comments/how often you are willing to rotate the token. For a longer-term solution, consider installing the Trunk GitHub App.
The resource owner should be the GitHub Organization or user that owns the appropriate repositories. See note on GitHub Org Ownership settings.
Select the repositories you wish to enable comments on.
Permissions - you must enable Issues (Read and write) and Pull requests (Read and write). Note: It is expected that metadata permissions automatically change.
If everything looks good, scroll down to double check that your Overview for permissions looks something like the image below. If so, create the token.
Once the token is generated, go back to the Trunk App (app.trunk.io) > click on your profile > Settings > Manage (under Organization) > Organization GitHub Token and enter the copied token into the text field, then finally press Submit.
You should see comments posted by your service account on your next PR.
Disable commenting
Pull Request comments are enabled by default. If you wish to disable the comments, you can do so by navigating to the Trunk App > click on your profile > Settings > RepoName. In the Flaky Tests heading, toggle the Summary Flaky Tests Reports setting.
Troubleshooting
GitHub Organization ownership
If you wish to set the resource owner to be a GitHub Organization, you should double check that this is allowed by navigating to your GitHub Organization > Settings > Personal access tokens > Settings. Make sure under "Fine-grained personal access tokens", you have "Allow access via fine-grained personal access tokens" selected.
Once the token is created, the Organization admin may need to approve the request for the token. This can be done by going to Github Organization > Settings > Personal access tokens > Pending requests. To confirm that the token was set, you should be able to see it under Active tokens.
At any point, feel free to reach out to our team through Slack.
Last updated