Getting Started

You'll need

• a monorepo on GitHub or GitLab — tests and code live in the same repository. Flakiness.io keys results by commit, so test results and the code that produced them must share the same git history.
• a CI workflow that runs your tests (GitHub Actions, GitLab CI/CD, or any other provider).

You'll have

• your first report visible on the Flakiness.io dashboard, with a GitHub check posted on new pull requests (GitHub-linked projects only).

1. Sign in to Flakiness.io with GitHub or GitLab.

   Choose the provider that hosts your repositories. You can connect the other provider later from your user settings and use either one to sign in.

   Note about “Act on your behalf”

   The Flakiness.io GitHub App only asks for read-only permissions. There is a long thread on GitHub’s community forum discussing the “Act on your behalf” wording here.

   Sign in with GitHub Sign in with GitLab

2. Create an Organization.

   Organization

   An Organization is a workspace that contains projects. Members share access control and billing. Think of it as your team or company account. See Access Control.

   Create Organization

3. Connect your repository. Flakiness.io reads your commit history to line test results up with the code that produced them, so it needs read access to the repository.

   GitHub

   Install the Flakiness.io GitHub App. It grants read-only access to commit history and permission to post PR checks.

   Install the GitHub App

   Then create a Project inside your organization and link it to the repository. Installing the App on its own connects nothing: the Project is what ties a repository to Flakiness.io.

   GitLab

   Create a GitLab personal access token with the read_api scope. Flakiness.io uses it to read commits and merge requests.

   Create a GitLab token

   Then create a Project inside your organization, choose GitLab, paste the token, and pick your project from the list.

   gitlab.com only

   Self-managed GitLab instances aren’t supported yet. The token needs Reporter access or above on the project, otherwise it won’t appear in the list.

   Project

   A Project maps 1:1 to a repository. It owns the test history, analytics, and settings for that repo.

4. Copy your flakinessProject identifier. Open your project page on Flakiness.io; the URL looks like https://flakiness.io/<orgSlug>/<projectSlug>. Your identifier is that <orgSlug>/<projectSlug> pair — you’ll paste it into the reporter config in the next step.

   What is flakinessProject?

   The orgSlug/projectSlug identifier tells the reporter which project to upload results to. For example, a project at https://flakiness.io/my-org/my-app has flakinessProject: 'my-org/my-app'.

5. Upload test results from CI. Pick your runner — each tile shows the install, config, and CI integration you need.

   Playwright Pytest Vitest Jest CucumberJS Bun Rust JUnit XML C Custom

   Install the reporter:

   npm install -D @flakiness/playwright

   Add it to your config:

   playwright.config.ts

   import { defineConfig } from '@playwright/test';
   
   export default defineConfig({
     reporter: [['@flakiness/playwright', {
       flakinessProject: 'my-org/my-app'
     }]],
   });

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: Playwright Test supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx playwright install --with-deps
         - run: npx playwright test

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN with your project as its audience: Playwright Test supports GitLab OIDC and uses flakinessProject for secretless authentication.

   .gitlab-ci.yml

   test:
     image: node:20
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - npm ci
       - npx playwright install --with-deps
       - npx playwright test

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other CI

   Expose an access token via FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx playwright test

   Install the reporter:

   uv add --dev pytest-flakiness
   # or: pip install pytest-flakiness

   The reporter auto-activates once installed.

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission and specify a FLAKINESS_PROJECT to upload the report to: Pytest supports Github OIDC for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       env:
         FLAKINESS_PROJECT: my-org/my-app
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with:
             python-version: '3.12'
         - run: pip install -e '.[test]'
         - run: pytest

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN and specify a FLAKINESS_PROJECT to upload the report to: Pytest supports GitLab OIDC for secretless authentication.

   .gitlab-ci.yml

   test:
     image: python:3.12
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     variables:
       FLAKINESS_PROJECT: my-org/my-app
     script:
       - pip install -e '.[test]'
       - pytest

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> pytest

   Install the reporter:

   npm install -D @flakiness/vitest

   Add it to your config:

   vitest.config.ts

   import { defineConfig } from 'vitest/config';
   import flakinessReporter from '@flakiness/vitest';
   
   export default defineConfig({
     test: {
       reporters: ['default', flakinessReporter({
         flakinessProject: 'my-org/my-app'
       })],
     },
   });

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: Vitest supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx vitest run

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN with your project as its audience: Vitest supports GitLab OIDC and uses flakinessProject for secretless authentication.

   .gitlab-ci.yml

   test:
     image: node:20
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - npm ci
       - npx vitest run

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx vitest run

   Install the reporter:

   npm install -D @flakiness/jest

   Add it to your config:

   jest.config.ts

   import type { Config } from 'jest';
   
   const config: Config = {
     testLocationInResults: true,
     reporters: [
       'default',
       ['@flakiness/jest', { flakinessProject: 'my-org/my-app' }],
     ],
   };
   
   export default config;

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: Jest supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx jest

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN with your project as its audience: Jest supports GitLab OIDC and uses flakinessProject for secretless authentication.

   .gitlab-ci.yml

   test:
     image: node:20
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - npm ci
       - npx jest

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx jest

   Install the reporter:

   npm install -D @flakiness/cucumberjs

   Add it to your config:

   cucumber.mjs

   export default {
     default: {
       format: ['@flakiness/cucumberjs'],
       formatOptions: {
         flakinessProject: 'my-org/my-app',
       },
     },
   };

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: CucumberJS supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx cucumber-js

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN with your project as its audience: CucumberJS supports GitLab OIDC and uses flakinessProject for secretless authentication.

   .gitlab-ci.yml

   test:
     image: node:20
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - npm ci
       - npx cucumber-js

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx cucumber-js

   bun test has built-in JUnit XML output. Run it with --reporter=junit, then use @flakiness/junit-xml to convert and upload the report.

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: the @flakiness/junit-xml step uses GitHub OIDC for secretless authentication. GitHub Actions ships with npx, so it runs without a separate install step.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: oven-sh/setup-bun@v2
         - run: bun install
         - run: bun test --reporter=junit --reporter-outfile=./junit.xml
         - name: Upload to Flakiness.io
           if: always()
           run: npx @flakiness/junit-xml ./junit.xml --category bun --flakiness-project my-org/my-app

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN: the @flakiness/junit-xml step uses GitLab OIDC for secretless authentication. It runs in after_script, which executes even when tests fail.

   .gitlab-ci.yml

   test:
     image: oven/bun:1
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - bun install
       - bun test --reporter=junit --reporter-outfile=./junit.xml
     after_script:
       - bunx @flakiness/junit-xml ./junit.xml --category bun --flakiness-project my-org/my-app

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other: Mac / Linux

   Install the standalone flakiness-junit-xml binary (no Node.js required), set FLAKINESS_ACCESS_TOKEN (from your project's settings), run your tests, then convert and upload:

   curl -fsSL https://github.com/flakiness/junit-xml/releases/latest/download/install.sh | sh
   bun test --reporter=junit --reporter-outfile=./junit.xml
   FLAKINESS_ACCESS_TOKEN=<token> flakiness-junit-xml ./junit.xml --category bun --flakiness-project my-org/my-app

   Other: Windows

   Install the standalone flakiness-junit-xml binary (no Node.js required), set FLAKINESS_ACCESS_TOKEN (from your project's settings), run your tests, then convert and upload:

   irm https://github.com/flakiness/junit-xml/releases/latest/download/install.ps1 | iex
   bun test --reporter=junit --reporter-outfile=./junit.xml
   $env:FLAKINESS_ACCESS_TOKEN = "<token>"
   flakiness-junit-xml ./junit.xml --category bun --flakiness-project my-org/my-app

   Rust doesn’t have a native Flakiness reporter. The simplest path is cargo-nextest, a drop-in cargo test replacement that emits JUnit XML.

   Adopt cargo-nextest. If your project doesn’t already use it, install it locally and switch your test command from cargo test to cargo nextest run. See the nextest installation guide for the recommended install for your OS.

   Configure the ci profile to emit JUnit XML:

   .config/nextest.toml

   [profile.ci.junit]
   path = "junit.xml"

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: the @flakiness/junit-xml step uses GitHub OIDC for secretless authentication. GitHub Actions ships with npx, so it runs without a separate install step.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: dtolnay/rust-toolchain@stable
         - run: cargo nextest run --profile ci
         - name: Upload to Flakiness.io
           if: always()
           run: npx @flakiness/junit-xml ./target/nextest/ci/junit.xml --category rust --flakiness-project my-org/my-app

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN: the upload uses GitLab OIDC for secretless authentication. The Rust image ships no Node.js, so install the standalone flakiness-junit-xml binary in after_script, which executes even when tests fail.

   .gitlab-ci.yml

   test:
     image: rust:latest
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - cargo nextest run --profile ci
     after_script:
       - curl -fsSL https://github.com/flakiness/junit-xml/releases/latest/download/install.sh | sh
       - flakiness-junit-xml ./target/nextest/ci/junit.xml --category rust --flakiness-project my-org/my-app

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other: Mac / Linux

   Install the standalone flakiness-junit-xml binary (no Node.js required), set FLAKINESS_ACCESS_TOKEN (from your project's settings), run your tests, then convert and upload:

   curl -fsSL https://github.com/flakiness/junit-xml/releases/latest/download/install.sh | sh
   cargo nextest run --profile ci
   FLAKINESS_ACCESS_TOKEN=<token> flakiness-junit-xml ./target/nextest/ci/junit.xml --category rust --flakiness-project my-org/my-app

   Other: Windows

   Install the standalone flakiness-junit-xml binary (no Node.js required), set FLAKINESS_ACCESS_TOKEN (from your project's settings), run your tests, then convert and upload:

   irm https://github.com/flakiness/junit-xml/releases/latest/download/install.ps1 | iex
   cargo nextest run --profile ci
   $env:FLAKINESS_ACCESS_TOKEN = "<token>"
   flakiness-junit-xml ./target/nextest/ci/junit.xml --category rust --flakiness-project my-org/my-app

   For runners without a native reporter, produce JUnit XML, then use @flakiness/junit-xml to convert and upload it.

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: the @flakiness/junit-xml step uses GitHub OIDC for secretless authentication. GitHub Actions ships with npx, so it runs without a separate install step.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         # ... your existing test steps, producing JUnit XML in ./test-results ...
         - name: Upload to Flakiness.io
           if: always()
           run: npx @flakiness/junit-xml ./test-results --flakiness-project my-org/my-app

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   GitLab CI/CD

   Declare an ID token named FLAKINESS_ID_TOKEN: the @flakiness/junit-xml step uses GitLab OIDC for secretless authentication. It runs in after_script, which executes even when tests fail.

   .gitlab-ci.yml

   test:
     id_tokens:
       FLAKINESS_ID_TOKEN:
         aud: my-org/my-app
     script:
       - ./run-tests.sh   # your existing test command, producing JUnit XML in ./test-results
     after_script:
       - npx @flakiness/junit-xml ./test-results --flakiness-project my-org/my-app

   For self-managed GitLab and other details, see GitLab CI/CD Setup.

   Other: Mac / Linux

   Install the standalone flakiness-junit-xml binary (no Node.js required), set FLAKINESS_ACCESS_TOKEN (from your project's settings), run your tests, then convert and upload:

   curl -fsSL https://github.com/flakiness/junit-xml/releases/latest/download/install.sh | sh
   <your test command producing JUnit XML in ./test-results>
   FLAKINESS_ACCESS_TOKEN=<token> flakiness-junit-xml ./test-results --flakiness-project my-org/my-app

   Other: Windows

   Install the standalone flakiness-junit-xml binary (no Node.js required), set FLAKINESS_ACCESS_TOKEN (from your project's settings), run your tests, then convert and upload:

   irm https://github.com/flakiness/junit-xml/releases/latest/download/install.ps1 | iex
   <your test command producing JUnit XML in ./test-results>
   $env:FLAKINESS_ACCESS_TOKEN = "<token>"
   flakiness-junit-xml ./test-results --flakiness-project my-org/my-app

   No official reporter for your runner? You have three options — see Custom Integration.

6. Push a commit and open the dashboard. Your first report should appear within seconds of the test job finishing.

   Tip

   If the upload didn’t happen, check your CI logs. The reporter prints a confirmation line on success ([flakiness.io] ✓ Uploaded as ...) and a descriptive error on failure.

Next steps

• Pull & Merge Requests — inspect regressions and, on GitHub, interpret the PR check.
• Slack Notifications — daily alerts on failures and regressions.
• Sharded Tests — configure CI matrix sharding.
• AI Agents — let Claude Code / Codex / Cursor investigate flaky tests.