Acceptable Flakiness

Flakiness.io provides two complementary tools for managing test flakiness: the Acceptable Flakiness Ratio setting and the Flip Rate column.

Acceptable Flakiness Ratio

The Acceptable Flakiness Ratio is a project-level threshold that defines your tolerance for flaky tests. It determines when flakiness impacts the Aggregated Status.

When Flakiness.io computes the Aggregated Status for a report, it checks how many tests flaked relative to the total number of tests. If the percentage exceeds your threshold, the status is marked as Flaked instead of Passed.

For example, with a ratio set to 5%:

• 3 flaky tests out of 100 total (3%) → Passed
• 6 flaky tests out of 100 total (6%) → Flaked

This also affects project badges — a Flaked status turns the badge orange.

Configuration

Go to Project Settings and adjust the Acceptable Flaky Test Ratio slider. The range is 0% to 20% in 0.1% increments.

The default is 1.5%.

Common Configurations

• 0% (Zero Tolerance) — recommended for critical systems (e.g., payments, healthcare) where any instability is a blocker
• 0.5% – 1% (Mature) — best for stable projects with high-quality test suites
• 3% – 5% (Standard) — a common baseline for active projects with high velocity

Flip Rate Column

Flip rate measures how often a test alternates between passing and failing across consecutive commits. In the platform UI, flip rate is displayed as a percentage column in the tests table.

Sort by flip rate (descending) to quickly surface the most unreliable tests in your project. A test with 100% flip rate alternates every single commit — a strong signal of flakiness. A test with 0% flip rate is completely stable.

Tip

Combine flip rate sorting with FQL filters to narrow down to specific files or tags:

flip>0% f:e2e          # flaky tests in e2e files
flip>50% #critical     # highly flaky critical tests