Fail Rate

Fail rate measures how often a test’s terminal status on a timeline is failing. A commit counts as a failure only if the test ended red after all retries.

Fail rate answers a different question than flip rate:

• Flip rate captures volatility — how often the test changes its outcome between commits.
• Fail rate captures failure frequency — what share of commits ended in failure.

Use fail rate to find tests that consistently fail (or fail often enough to be a problem) regardless of whether they also sometimes pass. Use flip rate to find tests that alternate between passing and failing.

Formula

For a given test in a timeline:

Fail Rate = Failures / Invocations

Where:

• Invocations is the number of commits where the test ran (any status other than skipped).
• Failures is the number of commits where the test’s status was failed or regressed.

If a test was never invoked in the window, fail rate is not defined.

Note

Failures are counted from the test’s commit-level status after retries. Flakiness.io first computes a test status per commit, and only commits whose terminal status is failed or regressed count as failures. A flaked commit (failed then passed on retry) counts as one invocation with zero failures, because the test eventually passed.

History Modes

The dashboard computes fail rate over the bars in the test’s History column, so the result depends on the history mode:

• By commit — each bar is a commit, and failures are counted per commit as described above.
• By 3 hours, By day, By week — each bar is a time period, and its status is the worst outcome the test had in that period. Invocations and failures count periods instead of commits. A period counts as a failure if the test failed on any of its commits, so in By day mode fail rate is the share of days on which the test failed at least once.

Examples

Commit history	Failures	Invocations	Fail Rate
✅ ✅ ✅ ✅ ✅	0	5	0%
❌ ❌ ❌ ❌ ❌	5	5	100%
✅ ✅ ✅ ❌ ❌	2	5	40%
✅ ❌ ✅ ❌ ✅	2	5	40%
✅ 🟡 ✅ ✅ ✅	0	5	0%
❌ 🟡 ❌ ✅ ❌	3	5	60%

(🟡 = flaked: failed then passed on retry within the same commit.)

Note that the third and fourth rows have the same fail rate (40%) but very different flip rates — fail rate ignores ordering, flip rate doesn’t.

Filtering by fail rate

Use the fail> operator in FQL to narrow tests by their fail rate:

fail>0%         # Tests that have failed at least once on the timeline
fail>50%        # Tests that fail more than half the time
fail<=25        # Tests with fail rate at most 25%