Pytest

Demo

Report demo is available at https://flakiness.io/flakiness/pytest-flakiness

The official Flakiness.io reporter for pytest.

Installation

1. Install using uv (recommended):

   uv add --dev pytest-flakiness

   Or via standard pip:

   pip install pytest-flakiness

CI/CD Integration

Reports are automatically uploaded to Flakiness.io at the end of each test run when authentication is configured. If no credentials are present, the report is still generated locally and can be uploaded manually later.

When uploading succeeds, you’ll see a confirmation in your terminal:

PASSED [100%]
==============================
[Flakiness] Report uploaded: https://flakiness.io/your_org/your_proj/run/1
==============================

Upload Method	Status	Guide
GitHub OIDC v0.13.0+	Supported	GitHub Actions
Access Token	Supported	GitHub Actions
Other CI	Supported	Other CI Providers

Tip

GitHub OIDC is the recommended method — no secrets to manage. It requires the --flakiness-project option. See the GitHub Actions guide for setup instructions.

Usage

Once installed, simply run pytest. The reporter will automatically activate, aggregate test results, and create Flakiness Report in the flakiness-report directory.

pytest

The generated report can be viewed interactively via the Flakiness CLI:

flakiness show

Tip

Make sure to add flakiness-report directory to your .gitignore:

flakiness-report/

Configuration Options

All options can be set via environment variables or command-line flags:

Flag	Environment Variable	Description
--flakiness-name	FLAKINESS_NAME	Name for this environment. Defaults to pytest
--flakiness-output-dir	FLAKINESS_OUTPUT_DIR	Local directory to save JSON report. Defaults to flakiness-report
--flakiness-project	FLAKINESS_PROJECT	Your Flakiness.io project in orgSlug/projectSlug format (required for GitHub OIDC)
--flakiness-access-token	FLAKINESS_ACCESS_TOKEN	Your Flakiness.io access token (required for upload)
--flakiness-endpoint	FLAKINESS_ENDPOINT	Flakiness.io service endpoint. Defaults to https://flakiness.io

Custom Environment Data

You can add custom metadata to your test runs using FK_ENV_* environment variables:

export FK_ENV_GPU_TYPE="H100"
export FK_ENV_DEPLOYMENT="staging"
pytest

The FK_ENV_ prefix is removed and keys are lowercased, e.g. FK_ENV_DEPLOYMENT becomes deployment, and FK_ENV_GPU_TYPE becomes gpu_type.

License & Source Code

The source code is available under MIT license at https://github.com/flakiness/pytest-flakiness