Suites & tasks

A suite is tasks × agents × repetitions. A run is one cell of that matrix, fully recorded.

suite: my-suite            # name (artifact directory / display)
runs_per_task: 3           # k — agents are non-deterministic; score pass@k
agents: [browser-use-gpt, claude-cu, gemini-cu, openai-cua]
tasks:
  - id: checkout                     # unique within the suite
    start_url: "https://…"           # where the agent begins
    goal: "…"                        # natural-language objective
    oracle:                          # how success is decided
      type: url-regex                #   url-regex | dom-selector | llm-judge | none
      pattern: "order-confirmed"
    policies: ["never create an account"]   # hard rules injected into the agent
    budget:
      max_steps: 40                  # hard stops; enforced server-side
      max_usd: 2.00
      max_minutes: 10

Oracles & verdicts

1. Programmatic oracle (url-regex, dom-selector) — deterministic; verdicts labeled oracle:* ("verified").
2. LLM judge (default) — WebJudge-style: final screenshot + full action log, strict about the "illusion of progress". Labeled llm-judge ("judged").
3. Failures get a failure class: perception | grounding | planning | execution | loop | auth-wall | captcha | timeout | error.

Friction telemetry (per run, automatic)

Dead clicks (click with no DOM reaction in 1.2s), retried actions, per-page funnel cost (steps/seconds/tokens/$), console & network errors joined to the timeline, bot-defense fingerprints (Cloudflare, DataDome, PerimeterX, reCAPTCHA, hCaptcha, Kasada, Arkose, Akamai).