Fail Rate

Fail rate measures how often a test’s terminal status on a timeline is failing. A commit counts as a failure only if the test ended red after all retries.

Fail rate answers a different question than flip rate:

• Flip rate captures volatility — how often the test changes its outcome between commits.
• Fail rate captures failure frequency — what share of commits ended in failure.

Use fail rate to find tests that consistently fail (or fail often enough to be a problem) regardless of whether they also sometimes pass. Use flip rate to find tests that alternate between passing and failing.

Formula

For a given test in a timeline:

Fail Rate = Failures / Invocations

Where:

• Invocations is the number of commits where the test ran (any status other than skipped).
• Failures is the number of commits where the test’s status was failed or regressed.

If a test was never invoked in the window, fail rate is not defined.

Note

Failures are counted from the test’s commit-level status after retries. Flakiness.io first computes a test status per commit, and only commits whose terminal status is failed or regressed count as failures. A flaked commit (failed then passed on retry) counts as one invocation with zero failures, because the test eventually passed.

Examples

| Commit history | Failures | Invocations | Fail Rate | |---|---|---|---| | ✅ ✅ ✅ ✅ ✅ | 0 | 5 | 0% | | ❌ ❌ ❌ ❌ ❌ | 5 | 5 | 100% | | ✅ ✅ ✅ ❌ ❌ | 2 | 5 | 40% | | ✅ ❌ ✅ ❌ ✅ | 2 | 5 | 40% | | ✅ 🟡 ✅ ✅ ✅ | 0 | 5 | 0% | | ❌ 🟡 ❌ ✅ ❌ | 3 | 5 | 60% |

(🟡 = flaked: failed then passed on retry within the same commit.)

Note that the third and fourth rows have the same fail rate (40%) but very different flip rates — fail rate ignores ordering, flip rate doesn’t.

Filtering by fail rate

Use the fail> operator in FQL to narrow tests by their fail rate:

fail>0%         # Tests that have failed at least once on the timeline
fail>50%        # Tests that fail more than half the time
fail<=25        # Tests with fail rate at most 25%