Test Regressions

If a test with a perfect execution history suddenly fails, it sends a strong signal that the code change is breaking the system.

Flakiness.io detects such situations and reports them by introducing a fifth test status - regression:

Classifying Failures as Regressions

To compute regressions for any timeline, Flakiness.io:

1. Computes test statuses for all commits
2. Classifies commit failures with a perfect recent execution history as regressions
3. Classifies failed runs inside Regressed commits as regressions
4. Classifies failed days with regressed commits inside as regressions

Tip

Flakiness.io currently inspects the last 10 tested commits for each test to determine whether it had a perfect execution history. The count is per-test: only commits where the test was actually observed count toward the 10.

Note

In other words, Flakiness.io converts certain ❌ in commits into 🔥, and then computes day and run regressions based on commit regressions.

To illustrate the logic, let’s say we have a testNervousSquirrel test which hasn’t been failing since forever, but suddenly failed yesterday in Commit X on May 25, 2025, and was failing since then:

Regressions in Commits

Flakiness.io classifies commit-level failures as regressions if the test has a perfect recent execution history.

So in our example:

• Since Commit X failed, and the test had a perfect recent execution history, the failure in Commit X will be classified as regression
• The test continued failing in both Commit Y and Commit Z. However, these failures are not new, so they will not be classified as regressions.

Regressions in Runs

Flakiness.io classifies a run-level failure as a regression if this failure is classified as regression on the commit level.

So if Commit X had, for example, 2 failing runs, then these run failures would be classified as regressions:

Regressions in Days

Flakiness.io classifies a “failed” status for a day as a “regression” if the regression happened during this day.

So in our example:

• Failure of the May 25, 2025 will be classified as a regression, since the test regressed during this day
• The failure on the following day of May 26, 2025, will not be classified as regression.