Flip Rate

Flip rate measures how often a test changes its outcome between consecutive commits in a timeline. A “flip” is any transition where a test switches from passing to failing, or from failing to passing.

Flip rate is a well-established metric for quantifying test flakiness. Apple’s research paper “Modeling and Ranking Flaky Tests at Apple” (ICSE-SEIP 2020) uses flip rate as one of two core signals for ranking flaky tests, achieving near-perfect accuracy in identification and alignment with human interpretation of flakiness.

Formula

For a given test in a timeline:

Flip Rate = Flips / (Invocations - 1)

Where:

• Invocations is the number of commits where the test was executed, with flaked commits counted twice
• Flips is the number of times the test’s status changed between consecutive commits

If a test has only 1 invocation, its flip rate is 0%. If a test has no invocations, the flip rate is not defined.

Note

Flips are counted between commits, not between individual runs within the same commit. Flakiness.io first computes a test status per commit, and then counts flips between those commit-level statuses.

A flaked commit (failed, then passed on retry) is the one exception. Flakiness.io counts it as a failure followed by a pass: two invocations and one flip for the retry, plus one more flip if the previous commit passed. A test whose only commit flaked has a flip rate of 100%.

History Modes

The dashboard computes flip rate over the bars in the test’s History column, so the result depends on the history mode:

• By commit — each bar is a commit, and flips are counted between commits as described above.
• By 3 hours, By day, By week — each bar is a time period, and its status is the worst outcome the test had in that period. Invocations and flips count periods instead of commits, and a period whose worst outcome is flaked counts the same way as a flaked commit. A test that failed on one commit and passed on a later commit of the same day gets a single failed bar for that day, so in By day mode that change is not a flip.

Why Order Matters

Unlike simple failure rate, flip rate captures the temporal pattern of test results. Two tests can have the exact same number of passes and failures but very different flip rates depending on the order of results.

Consider two tests, both with 3 passes and 2 failures across 5 commits:

Test A — failures are clustered together:

Commit 1	Commit 2	Commit 3	Commit 4	Commit 5
✅ passed	✅ passed	✅ passed	❌ failed	❌ failed

Transitions: ✅→✅, ✅→✅, ✅→❌, ❌→❌ — 1 flip out of 4 transitions = 25%

This pattern suggests a real regression rather than flakiness — the test was stable, then something broke it.

Test B — failures are scattered:

Commit 1	Commit 2	Commit 3	Commit 4	Commit 5
✅ passed	❌ failed	✅ passed	❌ failed	✅ passed

Transitions: ✅→❌, ❌→✅, ✅→❌, ❌→✅ — 4 flips out of 4 transitions = 100%

This pattern is a strong signal of a flaky test — the outcome keeps alternating regardless of code changes.

More Examples

Commit history	Flips	Invocations	Flip Rate
✅ ✅ ✅ ✅ ✅	0	5	0%
❌ ❌ ❌ ❌ ❌	0	5	0%
✅ ✅ ✅ ❌ ❌	1	5	25%
✅ ❌ ✅ ✅ ✅	2	5	50%
✅ ❌ ✅ ❌ ✅	4	5	100%
✅ 🟡 ✅ ✅ ✅	2	6	40%

(🟡 = flaked: failed then passed on retry within the same commit.)

A consistently passing or failing test has a flip rate of 0%. A test that alternates every commit has a flip rate of 100%.