Configuration

Flakiness.io requires environment variables for:

1. S3-compatible storage for report data
2. PostgreSQL database for application data
3. At least one sign-in provider: a GitHub App or GitLab OAuth application
4. Core application settings

Create a flakiness.env file with the following structure:

# S3 configuration
S3_ENDPOINT="http://..."
S3_ACCESS_KEY_ID="..."
S3_SECRET_ACCESS_KEY="..."
S3_REGION="..."
S3_BUCKET_NAME=flakiness-data

# PostgreSQL configuration
PGHOST="..."
PGPORT=5432
PGUSER="..."
PGPASSWORD="..."
PGDATABASE="..."
DB_ENCRYPTION_KEY="fkdb_..."

# GitHub App (optional when GitLab OAuth is configured)
GITHUB_APP_ID="..."
GITHUB_APP_PRIVATE_KEY="..."
GITHUB_APP_CLIENT_ID="..."
GITHUB_APP_CLIENT_SECRET="..."
GITHUB_APP_CALLBACK_URL="..."
GITHUB_APP_PUBLIC_URL="..."

# GitLab OAuth application (optional when a GitHub App is configured)
GITLAB_ORIGIN="https://gitlab.example.com"
GITLAB_APP_CLIENT_ID="..."
GITLAB_APP_CLIENT_SECRET="..."
GITLAB_APP_CALLBACK_URL="https://flakiness.example.com/login/gitlab/callback"

# Flakiness.io core
SITE_URL="https://flakiness.example.com"
COOKIE_DOMAIN="flakiness.example.com"
FLAKINESS_JWT_SECRET="..."
PORT=3000
SUPERUSERS="github:123456,gitlab:789012" # Comma-separated provider account IDs
FLAKINESS_LICENSE="your-license-key"

S3-compatible Storage

Flakiness.io stores test reports in an S3-compatible blob storage. Supported providers include:

• Amazon S3
• Google Cloud Storage
• Microsoft Azure Blob Storage
• Cloudflare R2
• Self-hosted MinIO

Create a bucket (e.g., flakiness-data) and configure these variables with your provider’s credentials:

S3_ENDPOINT="..."        # e.g., https://s3.amazonaws.com
S3_ACCESS_KEY_ID="..."
S3_SECRET_ACCESS_KEY="..."
S3_REGION="..."         # e.g., us-east-1
S3_BUCKET_NAME=flakiness-data

Note

Make sure to configure your bucket to allow CORS access from https://trace.playwright.dev. This is required to display Playwright Test traces.

PostgreSQL Database

Flakiness.io uses PostgreSQL (v16+) to store application data. The database size primarily scales with the number of users, not the number of test reports. A small instance (500MB, 1 vCPU) is typically sufficient.

Generate an encryption key for sensitive data:

docker run --rm -it cr.flakiness.io/app ./server/lib/cli.js create-database-encryption-key

This outputs a key prefixed with “fkdb_”, like:

fkdb_fd034105159d4cbde13ae19bf0b07298e0d53c0a5c05ba8ffc5af3c43960db10

Configure the database connection:

PGHOST="..."           # Database hostname
PGPORT=5432           # PostgreSQL port
PGUSER="..."          # Database user
PGPASSWORD="..."      # Database password
PGDATABASE="..."      # Database name
DB_ENCRYPTION_KEY="fkdb_..."  # Generated encryption key

GitHub App Configuration

Create a GitHub App for authentication following these steps:

1. Go to your GitHub organization settings or personal settings

2. Navigate to Developer Settings → GitHub Apps → New GitHub App

3. Configure the app:

   • Name: flakiness.io@YOUR_CORP (or similar)
   • Description: “flakiness.io aggregates test reports to display analytics information”
   • Homepage URL: Your SITE_URL
   • Callback URL: <SITE_URL>/login/github/callback
   • Expire user authorization tokens: ✅
   • Request user authorization during installation: Leave unchecked
   • Enable Device Flow: Leave unchecked
   • Webhook: Disable
   • Repository Permissions:
     • Contents: Read-only
     • Metadata: Read-only
   • Where can this GitHub App be installed?: Choose based on your needs
     • Your organization only: More restrictive
     • Any organization: More flexible

After creation, configure these environment variables:

GITHUB_APP_ID="..."              # From GitHub App settings
GITHUB_APP_PRIVATE_KEY="..."     # Generated in GitHub App settings
GITHUB_APP_CLIENT_ID="..."       # From GitHub App settings
GITHUB_APP_CLIENT_SECRET="..."   # From GitHub App settings
GITHUB_APP_CALLBACK_URL="https://flakiness.example.com/login/github/callback"
GITHUB_APP_PUBLIC_URL="..."      # GitHub App's public page URL

GitLab Configuration

Each Flakiness.io deployment connects to one GitLab origin. It defaults to https://gitlab.com; set GITLAB_ORIGIN to connect a self-hosted deployment to a self-managed GitLab instance instead. Configure the origin before users link accounts or projects:

GITLAB_ORIGIN="https://gitlab.example.com"

The origin may include a path prefix and must be reachable from the Flakiness.io container. If the instance uses an internal certificate authority, make that CA available to Node.js in the container, for example with NODE_EXTRA_CA_CERTS.

Create a user-owned or group-owned OAuth application on that same GitLab instance. See GitLab’s OAuth provider documentation for the current application setup flow.

Configure the application with:

• Redirect URI: <SITE_URL>/login/gitlab/callback
• Scope: read_user

GitLab displays the OAuth client ID as the Application ID. Add the application ID, secret, and exact redirect URI to your environment:

GITLAB_ORIGIN="https://gitlab.example.com"
GITLAB_APP_CLIENT_ID="..."        # Application ID from GitLab
GITLAB_APP_CLIENT_SECRET="..."    # Secret from GitLab
GITLAB_APP_CALLBACK_URL="https://flakiness.example.com/login/gitlab/callback"

The configured origin is used for sign-in, repository discovery and access, project links, and GitLab CI/CD OIDC verification. Changing it later does not migrate existing GitLab users or linked projects.

Core Application Settings

SITE_URL is the public URL where users access your deployment. Include the scheme and hostname, without a path:

SITE_URL="https://flakiness.example.com"

Flakiness.io uses this URL in Slack notifications, GitHub Checks, billing redirects, and links generated by the web application. It must match the URL configured in your reverse proxy and in each enabled provider application.

Set COOKIE_DOMAIN to the hostname that should receive authentication cookies:

COOKIE_DOMAIN="flakiness.example.com"

Prefix it with a dot only when cookies must be shared with subdomains, for example .flakiness.example.com.

1. Generate a JWT secret for user sessions:

docker run --rm -it cr.flakiness.io/app ./server/lib/cli.js create-jwt-token

2. Get GitHub IDs for administrators:

docker run --rm -it cr.flakiness.io/app ./server/lib/cli.js get-github-id USERNAME

3. Configure core settings:

SITE_URL="https://flakiness.example.com" # Public deployment URL
COOKIE_DOMAIN="flakiness.example.com"    # Cookie hostname, without scheme
FLAKINESS_JWT_SECRET="..."      # Generated JWT secret
PORT=3000                       # HTTP port for the service
SUPERUSERS="github:123456,gitlab:789012" # Comma-separated provider account IDs
FLAKINESS_LICENSE="..."         # Your license key

Running Flakiness.io

After configuring the environment variables, you can launch Flakiness.io using Docker. Mount the flakiness.env file into the container and start the application:

docker run \
  --rm \
  -p 3000:3000 \
  -v ./flakiness.env:/etc/flakiness/env \
  -it cr.flakiness.io/app:latest \
  node --env-file=/etc/flakiness/env ./server/lib/units/app_process.js

This command:

• Exposes port 3000 for web access
• Mounts the environment file
• Runs the latest version of the Flakiness.io container
• Starts the application process with the specified environment

Once the container and reverse proxy are running, access the Flakiness.io web interface at the URL configured in SITE_URL.

For a local-only deployment without a reverse proxy, use SITE_URL="http://localhost:3000" and COOKIE_DOMAIN="localhost", then navigate to http://localhost:3000.