Docs · CI analysis

CI auto-analysis

When CI runs fail, Exolar classifies the failure and takes the right action automatically: opening a fix PR, filing a bug, annotating a flake, or alerting on infrastructure issues.

Overview

Automated Triage

Every failure is classified into one of four categories so the right action fires without human intervention.

Ticket-Aware

Links failures to Linear / GitHub Issues tickets via branch name, commit message, or PR title.

Safe by Default

Guardrails enforce confidence thresholds, file-scope limits, and per-repo rate caps before any code changes land.

Flexible Trigger

Trigger via webhook, the REST API, or a Claude Code Action step · whatever fits your CI setup.

How It Works

The reporter uploads execution data after each run. On failure, the analysis engine classifies the root cause and dispatches the appropriate action.

CI Run → Reporter → Exolar → Analysis Engine → Action
                                   ├─ HEALABLE     → Auto-fix PR
                                   ├─ REAL_BUG     → GitHub Issue
                                   ├─ KNOWN_FLAKE  → Dashboard annotation
                                   └─ INFRA        → Alert only

CI Run – Your test suite runs in GitHub Actions (or any CI)
Reporter – The Exolar Playwright reporter uploads results, logs, and artifacts
Analysis Engine – LLM-assisted classification with historical flake data and ticket context
Action – The matched handler fires: PR, issue, annotation, or alert

Setup

1Install reporter with ticket-linking config

Add autoDetectTicket and autoHeal options to your reporter config. See Reporter docs for full options.

playwright.config.ts

// playwright.config.ts
import { exolar } from "@exolar-qa/playwright-reporter"

export default defineConfig({
  reporter: [
    ["html"],
    [exolar, {
      apiKey: process.env.EXOLAR_API_KEY,
      ticketMapping: {
        // Auto-detect ticket from branch name
        autoDetectTicket: true,
        pattern: /([A-Z]+-d+)/,          // e.g. ENG-1042
        source: "branch",                  // "branch" | "commit" | "pr-title"
      },
      autoHeal: {
        enabled: true,
        confidenceThreshold: 0.85,
      },
    }]
  ],
})

2Configure webhook

Go to Settings → Webhooks and add your endpoint. Subscribe to ci.run.failed events. Exolar signs every request with HMAC-SHA256 · see the Webhooks section below.

3Set up the analysis trigger

Choose between an OpenClaw sentinel agent (recommended for Montino infrastructure) or a GitHub Actions step:

trigger options

# Option A · OpenClaw sentinel agent (argus)
# In ~/.openclaw-nix/hosts/montino.nix, argus is already configured.
# Add the ci-analysis webhook URL to its trigger list.

# Option B · GitHub Actions (Claude Code Action)
- name: Trigger Exolar CI Analysis
  if: failure()
  uses: anthropics/claude-code-action@v1
  with:
    prompt: |
      Analyze the failed CI run via Exolar.
      POST https://app.exolar.dev/api/ci/analyze
      with run_id: ${{ github.run_id }}
    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

API Reference

Trigger analysis manually or from any CI step via the REST API. Authentication uses the same Bearer API key from Settings → API Keys.

POST /api/ci/analyze

POST /api/ci/analyze
Authorization: Bearer exolar_...
Content-Type: application/json

{
  "execution_id": "exec_abc123",   // optional · reporter execution ID
  "run_id": "run_gh_456"           // optional · GitHub Actions run ID
}

// Response
{
  "analysis_id": "anl_789",
  "classification": "HEALABLE",
  "confidence": 0.92,
  "action_plan": {
    "action": "open_pr",
    "title": "fix: update selector in checkout spec",
    "files": ["tests/checkout.spec.ts"],
    "patch": "...",
    "ticket": "ENG-1042"
  },
  "summary": "CSS selector outdated after design system update."
}

Field	Type	Description
`execution_id`	string?	Reporter execution ID · preferred when available
`run_id`	string?	GitHub Actions run ID · fallback if no execution_id
`analysis_id`	string	Unique ID for this analysis result
`classification`	enum	HEALABLE \| REAL_BUG \| KNOWN_FLAKE \| INFRA
`confidence`	number	0–1 score; actions only fire above configured threshold
`action_plan`	object	Action to take plus all data needed to execute it

execution_idstring?

Reporter execution ID · preferred when available

run_idstring?

GitHub Actions run ID · fallback if no execution_id

classificationenum

HEALABLE | REAL_BUG | KNOWN_FLAKE | INFRA

confidencenumber

0–1; actions only fire above configured threshold

Webhooks

Exolar sends a POST to your endpoint for each subscribed event. Every request includes an x-exolar-signature header for verification.

ci.run.failedFired when a CI run completes with at least one failed test

ci.run.fixedFired when a previously failing run passes again

ci.analysis.completeFired after the analysis engine produces a result

ci.heal.pr_openedFired when an auto-heal PR is opened

webhook verification

import crypto from "crypto"

function verifyWebhook(payload: string, signature: string, secret: string) {
  const expected = crypto
    .createHmac("sha256", secret)
    .update(payload)
    .digest("hex")
  return `sha256=${expected}` === signature
}

// In your handler:
const sig = req.headers["x-exolar-signature"]
if (!verifyWebhook(rawBody, sig, process.env.WEBHOOK_SECRET)) {
  return res.status(401).end()
}

Auto-Heal

When a failure is classified as HEALABLE with sufficient confidence, Exolar opens a draft PR with the suggested fix. Five fix strategies are supported:

Selector Update

Re-derives a stable CSS/ARIA selector when the DOM structure changed after a UI update.

Assertion Tolerance

Widens numeric or timing assertions that became brittle due to environment variance.

Wait Strategy

Replaces fixed sleeps or fragile waitForSelector calls with event-driven waits.

Data Fixture Refresh

Updates hardcoded test data that no longer matches the application's current state.

Import / Path Fix

Corrects broken imports or moved modules detected via static analysis of the error trace.

Bug Reports

Failures classified as REAL_BUG automatically open a GitHub Issue with full context: stack trace, failing test, artifact links, and the linked ticket. Deduplication prevents duplicate issues for the same root cause.

Deduplication

Before opening an issue, Exolar checks for existing open issues with a matching error signature. If a match is found, the new run is added as a comment instead.

Issue Content

Auto-generated issues include: test name, error message, reproduction steps, CI run link, artifact URLs, and the linked ticket from your ticketMapping config.

Linear Integration

Connect a Linear workspace in Settings → Integrations to enable ticket-aware analysis and acceptance criteria comparison.

ticketMapping

Configured in the reporter. Exolar extracts the ticket ID from the branch name, commit message, or PR title using the regex pattern you provide.

autoDetectTicket

When enabled, Exolar tries all three sources (branch, commit, pr-title) in order and uses the first match.

AC Comparison

When a ticket is resolved, the analysis engine compares the test coverage against the acceptance criteria from Linear. Gaps are flagged as missing coverage, not bugs.

Guardrails

Auto-heal actions are gated by multiple safety checks before any PR is opened.

Confidence Threshold

Default 0.85. Auto-heal only fires when the analysis confidence meets or exceeds the configured threshold. Adjust in reporter config.

File Scope Limit

A single auto-heal PR touches at most 3 files. Broader changes require human review and are flagged as REAL_BUG instead.

Rate Limit

At most 5 auto-heal PRs per repo per day to prevent runaway automation during a bad deploy.

Branch Protection Respected

Auto-heal PRs are always opened as drafts targeting a dedicated branch. They never push directly to main or protected branches.

Human-in-the-Loop Override

Set autoHeal.enabled: false in the reporter config to disable auto-heal globally while still receiving classifications and alerts.