LogoLogo
SlackLogin
  • Overview
    • Welcome
  • Setup & Configuration
    • Connecting to Trunk
    • Managing Your Organization
      • GitHub App Permissions
  • Integration with Slack
  • Flaky Tests (Beta)
    • Overview
    • Get Started
      • Test Frameworks
        • Android
        • Bazel
        • Behave
        • cargo-nextest
        • Cypress
        • Dart Test
        • Go
        • GoogleTest
        • Gradle
        • Jasmine
        • Jest
        • Karma
        • Maven
        • minitest
        • Mocha
        • Nightwatch
        • NUnit
        • Pest
        • PHPUnit
        • Playwright
        • Pytest
        • Robot Framework
        • RSpec
          • RSpec (Manual Uploads)
        • Swift Testing
        • Vitest
        • XCTest
        • Other Test Frameworks
      • CI Providers
        • Azure DevOps Pipelines
        • BitBucket Pipelines
        • Buildkite
        • CircleCI
        • Drone CI
        • GitHub Actions
        • GitLab
        • Jenkins
        • Semaphore CI
        • TeamCity
        • Travis CI
        • Other CI Providers
    • Dashboard
    • Flaky Test Detection
    • Quarantining
    • PR Comments
    • Ticketing Integrations
      • Jira Integration
      • Linear Integration
      • Other Ticketing Platforms
    • Webhooks
      • Slack Integration
      • Microsoft Teams Integration
      • GitHub Issues Integration
      • Linear Integration
    • Uploader CLI Reference
  • Merge Queue
    • Overview
    • How does it work?
    • Setup
      • Quick Start
      • Settings
      • Integration for Slack
    • Concepts and Optimizations
      • Predictive Testing
      • Optimistic Merging
      • Pending Failure Depth
      • Anti-Flake Protection
      • Batching
      • Parallel Queues
        • Bazel
        • Nx
        • API
      • FAQ
    • Priority
    • Managing Merge Queue
      • Using the Merge UI
      • Metrics
      • Command Line
    • Webhooks
    • Reference
  • Code Quality
    • Overview
    • Why Metalinters?
      • How does it work?
      • Why Code Quality?
    • Setup & Installation
      • Initialize Trunk
      • Local Linting
      • Linting in CI
      • Nightly Report (Deprecated)
      • OpenAI Codex Support
    • IDE Integration
      • VSCode
      • Neovim
      • GitHub Codespaces
    • Linters
      • Supported Linters
        • Actionlint
        • Ansible-lint
        • Autopep8
        • Bandit
        • Biome
        • Black
        • Brakeman
        • buf
        • Buildifier
        • cfnlint
        • Checkov
        • circleci
        • ClangFormat
        • clang-tidy
        • Clippy
        • cmake-format
        • codespell
        • cspell
        • cue-fmt
        • dart
        • deno
        • Detekt
        • djlint
        • dotenv-linter
        • dotnet-format
        • dustilock
        • ESLint
        • Flake8
        • git-diff-check
        • Gitleaks
        • Gofmt
        • gofumpt
        • goimports
        • gokart
        • golangci-lint
        • golines
        • google-java-format
        • graphql-schema-linter
        • hadolint
        • haml-lint
        • isort
        • iwyu
        • ktlint
        • kube-linter
        • markdown-link-check
        • markdown-table-prettify
        • Markdownlint
        • markdownlint-cli2
        • mypy
        • nancy
        • nixpkgs-fmt
        • opa
        • OSV-Scanner
        • Oxipng
        • perlcritic
        • perltidy
        • php-cs-fixer
        • phpstan
        • pmd
        • pragma-once
        • pre-commit-hooks
        • Prettier
        • prisma
        • psscriptanalyzer
        • Pylint
        • pyright
        • regal
        • remark-lint
        • renovate
        • rome
        • rubocop
        • Ruff
        • rufo
        • rustfmt
        • scalafmt
        • semgrep
        • ShellCheck
        • shfmt
        • sort-package-json
        • sourcery
        • sql-formatter
        • SQLFluff
        • sqlfmt
        • squawk
        • standardrb
        • stringslint
        • stylelint
        • stylua
        • SVGO
        • swiftformat
        • swiftlint
        • taplo
        • Terraform
        • terragrunt
        • terrascan
        • TFLint
        • tfsec
        • tofu
        • Trivy
        • Trufflehog
        • txtpbfmt
        • vale
        • Yamllint
        • yapf
      • Run Linters
      • Manage Linters
      • Configure Linters
      • Ignoring Issues and Files
      • Custom Linters
      • Shared Configs
      • Upgrades
    • Debugging
    • Licensing
  • CLI & API References
    • CLI Reference
      • Install
      • Getting Started
        • Code Quality
        • Merge Queue
        • Flaky Tests
        • Tools
        • Actions
          • Git Hooks
        • Announce
      • Compatibility
      • Caching
      • Commands Reference
        • Code Quality
        • Actions
        • Merge
      • Configuration
        • Plugins
          • Share Config Between Codebases
          • Exporting Linter Configs
        • Runtimes
        • Tools
        • Lint
          • Definitions
          • Commands
          • Output
          • Output Parsing
          • Files and Caching
          • Dependencies
          • Auto-Enable
        • Actions
          • Notifications
          • Logging and Troubleshooting
        • Merge
        • Telemetry
        • Per User Overrides
    • API Reference
      • Flaky Tests
      • Merge Queue
      • Webhooks Reference
  • Pricing & Security
    • Security
  • Billing
  • Community & Support
  • Links
    • Open App
    • Slack Community
    • Changelog
    • Feature Requests
On this page
  • Configuration
  • Disable commenting
  • Troubleshooting
Edit on GitHub
  1. Flaky Tests (Beta)

PR Comments

Flaky Tests provides summary analytics about tests running on Pull Requests

Last updated 3 months ago

Flaky Tests can post comments on GitHub pull requests to provide a summary of all the tests run on this PR, across multiple CI jobs or even CI systems, as well as whether those failures are due to flakiness, the history of the failures of that test, and more.

Each GitHub comment has a full summary report, containing all tests that passed, failed, flaked, skipped, and quarantined on this PR.

Each test case also contains the full stack trace attached when opened. You can also click on the Job run link which will take you to the full CI logs.

Configuration

Without Trunk GitHub App

It's recommended that the Trunk GitHub App be used to manage GitHub comments. If you need to generate comments without the Trunk GitHub app, you can do so with a service account and an API token.

  1. Create a dedicated GitHub SVC account (Service Account) with access to the repositories in your GitHub Organization that Flaky Tests will comment on e.g., trunk-analytics-user.

  2. Name the new token something memorable. ex: trunk-flaky-tests-token.

  3. The expiry time is up to you - however long you wish to try out Flaky Tests comments/how often you are willing to rotate the token. For a longer-term solution, consider installing the Trunk GitHub App.

  4. Select the repositories you wish to enable comments on.

  5. Permissions - you must enable Issues (Read and write) and Pull requests (Read and write). Note: It is expected that metadata permissions automatically change.

  6. If everything looks good, scroll down to double check that your Overview for permissions looks something like the image below. If so, create the token.

You should see comments posted by your service account on your next PR.

Disable commenting

Pull Request comments are enabled by default. If you wish to disable the comments, you can do so by navigating to the Trunk App > click on your profile > Settings > RepoName. In the Flakey Tests heading, toggle the Summary Flaky Tests Reports setting.

Troubleshooting

GitHub Org Ownership

If you wish to set the resource owner to be a GitHub Organization, you should double check that this is allowed by navigating to your GitHub Organization > Settings > Personal access tokens > Settings. Make sure under "Fine-grained personal access tokens", you have "Allow access via fine-grained personal access tokens" selected.

Once the token is created, the Organization admin may need to approve the request for the token. This can be done by going to Github Organization > Settings > Personal access tokens > Pending requests. To confirm that the token was set, you should be able to see it under Active tokens.

If you have the and are test results on pull requests, expect to start seeing comments on your Pull Requests soon. If you prefer not to use the Trunk GitHub App, you can still set up comments on your Pull Requests by providing Trunk with a GitHub access token.

On , for trunk-analytics-user (or whichever user you wish to use), generate a by navigating to Settings > Developer settings > Personal access token > Fine-grained tokens > Generate new token.

The resource owner should be the GitHub Organization or user that owns the appropriate repositories.

Once the token is generated, go back to the Trunk App () > click on your profile > Settings > Manage (under Organization) > Organization GitHub Token and enter the copied token into the text field, then finally press Submit.

At any point, feel free to reach out to our team .

Trunk GitHub App installed
uploading JUnit XML
github.com
Personal access token
app.trunk.io
through Slack
See note on GitHub Org Ownership settings.