Changelog

September 18, 2026

• Query a specific day from the CLI. flakiness list tests now accepts --day YYYY-MM-DD, so you can inspect tests from a past day. Dates use the project’s timezone, matching the dashboard calendar. It queries the default branch unless you also pass --branch.

  # Inspect a day on the default branch
  flakiness list tests --project myorg/myproject --day 2026-09-17
  
  # Inspect a day on a specific branch
  flakiness list tests --project myorg/myproject --branch feature/login --day 2026-09-17

  Without --day, the CLI still shows tests from the day of the branch’s head commit. --day cannot be combined with --pr. See the CLI commands reference.

September 11, 2026

• Pull request comments. Flakiness.io now posts a test results comment on every GitHub pull request. The comment opens with the most recently tested commit and its outcome counts, lists the regressed and failed tests with links into their reports, and keeps flaky tests in a collapsed section. The same comment is rewritten in place as new results arrive, so a pull request never collects a thread of stale summaries. Comments are on by default and can be turned off per project. See GitHub Reporting.

  

  A GitHub App installed before this release has no permission to write comments. Projects in that state show a banner linking to the approval page for the new Pull requests: Read and write permission. Comments start appearing on the next test run after an organization owner approves it.

• Faster dashboards. Projects now serve their data from a prepared snapshot instead of recomputing stats per request. Commit analysis and the daily outcomes computation were optimized separately on top of it.

August 28, 2026

• Outcome breakdown in the history chart. Hovering a day on the History chart now shows how that day’s tests split across Regressed, Failed, Flaked, Passed, and Skipped, with the same colors the calendar below it uses. The legend above the chart plots individual outcomes: pick Regressed to chart regressions alone, or clear the selection to go back to the day total.

August 8, 2026

• Self-managed GitLab. A self-hosted Flakiness.io deployment can now connect to a self-managed GitLab instance instead of gitlab.com. Set GITLAB_ORIGIN to the instance URL; it is used for sign-in, repository discovery and cloning, project links, and GitLab CI/CD OIDC verification. See Configuration.

July 30, 2026

• GitLab support. Flakiness.io now works with GitLab from sign-in through test analytics. Connect a GitLab repository to browse its commits and merge requests, sign in with GitLab or link it to an existing account, and use GitLab CI/CD OIDC for secretless report uploads. See Getting Started and GitLab CI/CD Setup.

  Users who connect their GitLab identity also receive Viewer access to private Flakiness projects whose repositories they can access with the Reporter role or higher. See Access Control.

• Vitest Browser Mode. @flakiness/vitest now captures Browser Mode failure screenshots and image diffs and uploads them with the test report.

• Flakiness.io Blog. We launched blog.flakiness.io for product updates, test engineering guides, and behind-the-scenes looks at how Flakiness.io is built.

• Account settings. Users can edit their profile, refresh their avatar from a connected provider, link or unlink GitHub and GitLab accounts, inspect and log out active browser sessions, and delete their Flakiness account.

• Better Playwright shard balancing. @flakiness/playwright now uses Playwright’s native reporter sharding API and prints the balancing plan before a run and the result after each shard finishes.

July 13, 2026

• GitHub Checks. Flakiness.io can post test results back to GitHub as check runs. It can now be turned off per project in Settings, separately for branch commits and for pull requests.

• Audit Log. Enterprise organizations can get an audit log of administrative changes: who changed a setting, removed a member, rotated a token, or deleted a project, and when each happened. Entries are kept for one year. Contact us to enable it for your organization.

• Full test and suite names. The tests table now shows complete suite and test names instead of shortening them, so long paths and JUnit-style fully-qualified names stay readable.

• Annotation highlighting in search. Filtering tests with FQL now highlights the matching text inside an annotation’s owner, type, or description, so it is clear why each row matched.

• Shard balancing for public projects and forks. The Durations API can now be read with viewer access. Runs on public projects, including pull requests from forks that carry no upload token, can now fetch predicted durations and balance shards.

June 22, 2026

• Daily test history. Test history can now be bucketed by day instead of by commit, with each bar standing for a whole day. A test’s flip rate, fail rate, and duration trend are then computed over days rather than commits, so sorting by flip rate or fail rate gives a long-term view of test performance instead of commit-to-commit, which might be noisy. Pick the default for a project in its settings (By commit or By day) and override it per view from the selector on the History column header; the flakiness CLI takes the same --history commit|day flag on list tests.

June 9, 2026

• Shard Balancing. Test runners can now fetch each test’s predicted duration from a new Durations API and split the suite into shards of roughly equal running time, so a run finishes sooner instead of waiting on its slowest shard. See Balancing Playwright Test Shards for how it works and its tradeoffs.

  Playwright Test is the first test runner to support it; see its documentation for setup. More runners are coming.

June 1, 2026

• Fail rate metric. A new fail rate metric shows what share of a test’s commits ended in failure after retries. Filter by it with the fail> operator in FQL. Where flip rate captures how often a test changes outcome, fail rate captures how often it ends red.
• Worst result in the test history calendar. Each day in a test’s history calendar now reflects its worst result that day, so days where the test failed stand out at a glance.

May 27, 2026

• Streamlined project creation. Creating a new project now is as easy as installing Flakiness.io app and picking your repository from the dropdown.
• Organization billing e-mails. Org administrators can now configure billing emails.
• User profile page. A dedicated profile page in user settings.

May 18, 2026

• Runtime & test runner versions. All Flakiness.io reporters now collect the runtime and test runner versions, and these are rendered as part of the run. Update your reporters to pick this up.
• Configurable table columns. Data tables now have a column-config cog to hide or show individual columns. Choices are remembered per browser.
• Open-source JUnit XML converter. The JUnit XML converter is now its own open-source package, @flakiness/junit-xml, which converts and uploads JUnit XML in a single step. The CLI convert-junit command is now deprecated. See the documentation.
• Bun support. bun test results can now be ingested into Flakiness.io via its built-in JUnit XML output. See the Bun example.
• Rust support. Rust test results can now be ingested into Flakiness.io via cargo-nextest’s JUnit XML output. See the Rust example.

May 8, 2026

• No more user limits. Plans no longer cap the number of users who can view a dashboard. Pricing is now structured around storage and integrations.
• Free plan. Public dashboards are now free, with storage sized to cover typical project workloads. Great fit for open source projects.
• GitHub-native access control. Authentication and authorization both flow through GitHub. A private dashboard for a private GitHub repository is automatically readable by anyone with access to that repository.

Apr 27, 2026

• Jest support. Flakiness.io Jest reporter is now available at github.com/flakiness/jest.

• Runs with no environments are now visible. When a test runner crashes early and reports results without any environment attached, those runs now appear in the dashboard instead of being silently dropped.

Apr 20, 2026

• Rich attachments. Test steps now render their own attachments inline, with first-class support for text (expandable), images, image diffs, HTML, video, and Playwright traces. Attachments are attributed to the step that produced them, making failure context much easier to scan.
• Quick jump to test history. A new “activity” icon next to test name opens that test’s full history in one click.
• Polish & fixes. a collection of assorted small UI cleanups

Apr 10, 2026

• Redesigned history tab. The History view has been rebuilt from the ground up to allow commit streaming. The new History tab shows test count and duration trends for the selected 3-month window.
• Commits get more context. Commits now display actual head references.
• Visual polish. All main screens were polished & de-cluttered. Enjoy the familiar but refreshing interactions!
• Open Source Program. Announcing the Flakiness.io Open Source program — any repository with 5,000+ stars on GitHub is eligible for a free Flakiness.io dashboard. Apply via email.

Apr 3, 2026

• Smarter regression history. Flakiness.io now tracks each test against its own recent commit history instead of a fixed calendar window. This makes regression detection more reliable for projects with infrequent activity and removes the regression window setting.
• PR branch labels. Pull requests now show their base and target branches, helping with more complicated branching workflows.
• Clearer environment timelines. Timeline rendering is now tied more closely to the selected environment, so reports and commit history stay aligned in projects that test across multiple environments.
• Project-local usage time. Usage views now follow the project’s configured timezone.
• Performance and reliability improvements. Dashboard loading is faster, and several history and pull-request edge cases were fixed behind the scenes.

Mar 27, 2026

• Nicer run view. Runs now display environment attributes and colored test-runner icons, making it easier to scan results at a glance.
• Run titles. Test runs can now carry a human-readable title — set it in your reporter config and it’ll appear in the Report Viewer header and in run lists on the platform. See Naming Runs.
• Token rotation. Flakiness Access Tokens can now be rotated from project settings, so you can cycle credentials without recreating the project.
• CLI: auth whoami → auth status. The command has been renamed to auth status (whoami is kept as an alias).
• New runs index. We’ve rebuilt how runs are indexed internally. This is a foundation for a set of upcoming features, including a redesigned history tab.

Mar 20, 2026

• Cucumber.js support. Flakiness.io now has an official reporter for CucumberJS — bringing BDD-style test suites into the fold alongside Playwright, Vitest, and pytest.
• TestNG support. The JUnit converter now understands TestNG XML reports, opening the door for Java teams using TestNG.
• Smarter agentic workflows. AI agents can now filter tests by environment, making investigation skills more targeted. Skills also install to your home directory by default for easier setup.
• CLI upload progress. The new --progress flag gives you real-time feedback when uploading large batches of reports.

Mar 16, 2026

Flakiness CLI v0.218+ introduces Agentic Workflows — AI coding agents can now query your test data, investigate flaky tests, and fix regressions autonomously.

Install a skill into your agent of choice:

flakiness skills install --agent <claude|codex|cursor>

Then just ask your agent to fix your tests:

• “Fix the failing tests in PR #42”
• “Find the most flaky tests in our project”
• “Which tests have been failing with timeout errors?”

Supports Claude Code, Codex, and Cursor. You can also set up a scheduled GitHub Actions cron job that automatically deflakes your test suite — no human intervention needed.

Full documentation: Agentic Workflows.

Mar 9, 2026

Initial release of the official Vitest reporter is now available: @flakiness/vitest.

Feb 24, 2026

• Projects can now configure Slack alerting via webhooks in their settings.
• pytest-flakiness now supports GitHub OIDC for uploads.

Feb 23, 2026

• Github OIDC is now supported for automation. @flakiness/playwright v1.1.0 is the first reporter to implement Github OIDC authentication. Configuration docs
• A richer view for the “Runs” page.
• Runs can now be deleted individually if needed.

Feb 20, 2026

• Runs page improvements: no more “unparsed runs”. If Flakiness.io cannot process a run, it now shows an alert icon with a suggestion to delete the malformed runs.
• Custom retention period for attachments can now be configured in the project’s usage page.

Feb 15, 2026

• Each Pull Request and commit now has a “run” button — a toggle that opens a dialog with all runs for the given commit or Pull Request.
• Runs are sortable by run ID, test outcomes, or start timestamp.