GitHub Reporting

Flakiness.io reports test results back to GitHub in two forms: a status check on a commit or pull request, and a comment on a pull request. Both are configured per project, in the GitHub Reporting section of the project’s Settings page.

There are three independent switches:

• Status checks for commits — post a check on every branch commit that has results
• Status checks for pull requests — post a check on every pull request that has results
• Pull request sticky comments — post and maintain a results comment on every pull request

All three are on by default, and project editors can change them.

Status checks

Flakiness.io posts a single check named Flakiness.io. Its title is the outcome breakdown for the commit, and its details link opens the corresponding report.

The check concludes one of three ways:

Conclusion	When
Success	Nothing regressed and nothing failed. Flaky and skipped tests still pass the check.
Failure	At least one test regressed or failed.
Neutral	The commit has reports, but they contain no tests.

Pull request comments

Flakiness.io maintains one comment per pull request with the results of its most recently tested commit.

The comment has 4 parts:

1. A summary table with the head commit, the overall verdict, the outcome counts, and the number of CI runs that reported results. Each count links into the report filtered to that outcome.
2. Regressed tests, listed with the environment they ran in. Every entry links directly to that test’s report.
3. Failed tests, in the same format.
4. Flaky tests, in the same format.

Each section lists at most ten tests and ends with a link to the full filtered list when there are more. Passed and skipped tests are counted in the summary table but not listed.

Comment Update

Flakiness.io rewrites the same comment in place rather than adding a new one, so a pull request never collects a thread of stale summaries. The rewrite happens when:

• the first results arrive for the pull request,
• a later CI run reports more results for the same commit,
• a new commit is pushed and tested.

The comment always reflects the most recently tested commit, which is not necessarily the pull request’s head. If the newest commit has not been tested yet, the comment keeps showing the last commit that was.