Pull Requests & Regressions

You'll need

• Getting Started complete
• a CI guide complete: GitHub Actions Setup or CI Configuration
• at least one PR opened against your default branch

You'll have

• confidence in reading the Flakiness.io check on any PR and deciding whether it’s mergeable.

Regression

A regression is a failing test that the PR is responsible for — either an existing test that was passing on the target branch and now fails, or a brand-new test that fails on first run. See Concepts → Regressions for the mechanics.

How It Works

When tests run on a pull request, Flakiness.io:

1. Collects test data from all test runs associated with the PR.
2. Compares results against the target branch history to classify each failure. An existing test with a perfect recent history on the target branch that now fails is a regression. A test that has been flaky or failing on the target branch is not.
3. Flags brand-new tests that fail (tests the PR adds that don’t pass) as regressions too.
4. Reports status back to GitHub via a Flakiness.io check on the pull request.

Check outcomes

The check badge has three outcomes:

• Mergeable — no failures and no regressions in this PR. Everything passes.
• Has Regressions — the PR either breaks a previously-passing test with a perfect history, or adds a new test that fails on first run. Investigate before merging.
• Neutral — there are failures, but the same tests have also been failing or flaking on the target branch recently. They might be noise, or they might be real — Flakiness.io can’t say for sure. Worth a look before merging.

Fix Regressions with AI Agents

If your project has the AI Agents skill installed, you can ask Claude Code, Codex, or Cursor to fix the regressions directly: “Fix the failing tests in PR #42 in myorg/myproject”. The agent will query Flakiness.io for the regressed tests, read their stack traces, and make the fix.

Pull Requests from Forks

Open-source projects that accept contributions from forks need a slightly different setup because GitHub doesn’t expose secrets or OIDC to fork-PR workflows. See GitHub Actions for Open Source Projects for the two-workflow pattern.