Configuration

Flakiness.io requires environment variables for:

1. S3-compatible storage for report data
2. PostgreSQL database for application data
3. GitHub App for authentication
4. Core application settings

Create a flakiness.env file with the following structure:

# S3 configuration
S3_ENDPOINT="http://..."
S3_ACCESS_KEY_ID="..."
S3_SECRET_ACCESS_KEY="..."
S3_REGION="..."
S3_BUCKET_NAME=flakiness-data

# PostgreSQL configuration
PGHOST="..."
PGPORT=5432
PGUSER="..."
PGPASSWORD="..."
PGDATABASE="..."
DB_ENCRYPTION_KEY="fkdb_..."

# Github App
GITHUB_APP_ID="..."
GITHUB_APP_PRIVATE_KEY="..."
GITHUB_APP_CLIENT_ID="..."
GITHUB_APP_CLIENT_SECRET="..."
GITHUB_APP_CALLBACK_URL="..."
GITHUB_APP_PUBLIC_URL="..."

# Flakiness.io core
SITE_URL="https://flakiness.example.com"
COOKIE_DOMAIN="flakiness.example.com"
FLAKINESS_JWT_SECRET="..."
PORT=3000
SUPERUSERS="123456,789012" # Comma-separated GitHub user IDs
DISABLE_ORG_CREATION=1
FLAKINESS_LICENSE="your-license-key"

S3-compatible Storage

Flakiness.io stores test reports in an S3-compatible blob storage. Supported providers include:

• Amazon S3
• Google Cloud Storage
• Microsoft Azure Blob Storage
• Cloudflare R2
• Self-hosted MinIO

Create a bucket (e.g., flakiness-data) and configure these variables with your provider’s credentials:

S3_ENDPOINT="..."        # e.g., https://s3.amazonaws.com
S3_ACCESS_KEY_ID="..."
S3_SECRET_ACCESS_KEY="..."
S3_REGION="..."         # e.g., us-east-1
S3_BUCKET_NAME=flakiness-data

Note

Make sure to configure your bucket to allow CORS access from https://trace.playwright.dev. This is required to display Playwright Test traces.

PostgreSQL Database

Flakiness.io uses PostgreSQL (v16+) to store application data. The database size primarily scales with the number of users, not the number of test reports. A small instance (500MB, 1 vCPU) is typically sufficient.

Generate an encryption key for sensitive data:

docker run --rm -it cr.flakiness.io/app ./server/lib/cli.js create-database-encryption-key

This outputs a key prefixed with “fkdb_”, like:

fkdb_fd034105159d4cbde13ae19bf0b07298e0d53c0a5c05ba8ffc5af3c43960db10

Configure the database connection:

PGHOST="..."           # Database hostname
PGPORT=5432           # PostgreSQL port
PGUSER="..."          # Database user
PGPASSWORD="..."      # Database password
PGDATABASE="..."      # Database name
DB_ENCRYPTION_KEY="fkdb_..."  # Generated encryption key

GitHub App Configuration

Create a GitHub App for authentication following these steps:

1. Go to your GitHub organization settings or personal settings

2. Navigate to Developer Settings → GitHub Apps → New GitHub App

3. Configure the app:

   • Name: flakiness.io@YOUR_CORP (or similar)
   • Description: “flakiness.io aggregates test reports to display analytics information”
   • Homepage URL: Your SITE_URL
   • Callback URL: <SITE_URL>/login/github/callback
   • Expire user authorization tokens: ✅
   • Request user authorization during installation: Leave unchecked
   • Enable Device Flow: Leave unchecked
   • Webhook: Disable
   • Repository Permissions:
     • Contents: Read-only
     • Metadata: Read-only
   • Where can this GitHub App be installed?: Choose based on your needs
     • Your organization only: More restrictive
     • Any organization: More flexible

After creation, configure these environment variables:

GITHUB_APP_ID="..."              # From GitHub App settings
GITHUB_APP_PRIVATE_KEY="..."     # Generated in GitHub App settings
GITHUB_APP_CLIENT_ID="..."       # From GitHub App settings
GITHUB_APP_CLIENT_SECRET="..."   # From GitHub App settings
GITHUB_APP_CALLBACK_URL="https://flakiness.example.com" # Same as SITE_URL
GITHUB_APP_PUBLIC_URL="..."      # GitHub App's public page URL

Core Application Settings

SITE_URL is the public URL where users access your deployment. Include the scheme and hostname, without a path:

SITE_URL="https://flakiness.example.com"

Flakiness.io uses this URL in Slack notifications, GitHub Checks, billing redirects, and links generated by the web application. It must match the URL configured in your reverse proxy and GitHub App.

Set COOKIE_DOMAIN to the hostname that should receive authentication cookies:

COOKIE_DOMAIN="flakiness.example.com"

Prefix it with a dot only when cookies must be shared with subdomains, for example .flakiness.example.com.

1. Generate a JWT secret for user sessions:

docker run --rm -it cr.flakiness.io/app ./server/lib/cli.js create-jwt-token

2. Get GitHub IDs for administrators:

docker run --rm -it cr.flakiness.io/app ./server/lib/cli.js get-github-id USERNAME

3. Configure core settings:

SITE_URL="https://flakiness.example.com" # Public deployment URL
COOKIE_DOMAIN="flakiness.example.com"    # Cookie hostname, without scheme
FLAKINESS_JWT_SECRET="..."      # Generated JWT secret
PORT=3000                       # HTTP port for the service
SUPERUSERS="123456,789012"      # Comma-separated GitHub IDs
DISABLE_ORG_CREATION=1          # Recommended: only admins can create orgs
FLAKINESS_LICENSE="..."         # Your license key

Running Flakiness.io

After configuring the environment variables, you can launch Flakiness.io using Docker. Mount the flakiness.env file into the container and start the application:

docker run \
  --rm \
  -p 3000:3000 \
  -v ./flakiness.env:/etc/flakiness/env \
  -it cr.flakiness.io/app:latest \
  node --env-file=/etc/flakiness/env ./server/lib/units/app_process.js

This command:

• Exposes port 3000 for web access
• Mounts the environment file
• Runs the latest version of the Flakiness.io container
• Starts the application process with the specified environment

Once the container and reverse proxy are running, access the Flakiness.io web interface at the URL configured in SITE_URL.

For a local-only deployment without a reverse proxy, use SITE_URL="http://localhost:3000" and COOKIE_DOMAIN="localhost", then navigate to http://localhost:3000.