# ArbiterQA

> ArbiterQA is a headless QA agent for webpages and emails: send a URL, raw HTML, or an email, pick which validations to run, and get back pass / fail / error per validation — with evidence (screenshots, structured findings, reasoning). Visual validations are judged by vision LLMs against real captures across viewports and mail clients; functional validations are deterministic code. Every validation is human-reviewed and gets more accurate over time.

Everything is plain HTTP — no SDK required.

**Important notes:**

- API base URL: `https://api.arbiterqa.com` — override with `ARBITER_API_URL` for staging or dev instances. The dashboard is `https://app.arbiterqa.com` (sign-in and UI only).
- Most `/api/*` endpoints take `Authorization: Bearer <key>`. Public discovery: `GET /api/validations`, `GET /api/validation-sets`, `GET /api/validation-sets/:setId`, and `GET /api/errors`; `POST /api/signup` is the public account door.
- Get a key with `npx arbiterqa login` (OAuth device flow; creates the account on first run — free to try). Read it back with `npx arbiterqa print-key`. Keys are also retrievable and rotatable in the dashboard under API keys.
- **No browser? Sign yourself up.** `POST /api/signup` with `{ "email": "you@example.com" }` returns `{ apiKey, keyId, customerId, orgCode, email }` — no auth, no device loop, and the same free account a human gets. Also the public MCP tool `create_account`. **Store `apiKey` immediately: it is shown once.** Use an address your user actually controls, because signing in with it later claims the account. An email that already has an account is refused (409 `ACCOUNT_EXISTS`) — sign in and mint a key instead.
- Discover individual checks with `GET /api/validations`. Discover published bundles with `GET /api/validation-sets`, then run one by sending its `id` as `validationSetId`; use `validations[]` for an explicit mix. Only ids with `runnable: true` are accepted in jobs; ids with `requiresConfiguration: true` need `parameters` on their validation entry or they are skipped.
- Human-readable catalog pages (public, no auth): `https://www.arbiterqa.com/validations`, `/validations/{slug}/standard`, and `/validations/{slug}/custom` when the check has Custom Parameters. Detail JSON: `GET /api/validations/{idOrSlug}`. Catalog dump with parameters: `GET /api/validations?include=parameters`. Sitemap: `/sitemap.xml`. Privacy Policy: `https://www.arbiterqa.com/privacy`. Terms of Service: `https://www.arbiterqa.com/terms`.
- **MCP (preferred when your host speaks it):** `https://api.arbiterqa.com/mcp` — streamable HTTP. Discovery tools (`search_validations`, `get_validation`, `recommend_checks`, `list_validations`, `list_validation_sets`, `get_validation_set`, …) and `create_account` need no key; estimate / run / account / feedback need `Authorization: Bearer <key>`. Protected-resource metadata: `GET /.well-known/oauth-protected-resource/mcp`.
- **Brand Card (SchemaFirst):** `GET /.well-known/brand-card.json` — logos, colours, type, voice for agents and design systems. Experimental category catalog: `/.well-known/brand-schema.json`.
- `url` and `static_html` jobs are synchronous — the `POST /api/jobs` response carries full results. `email` jobs are asynchronous: send one real email to the returned one-time `testEmail` address, then poll `GET /api/jobs`.
- Results are verdicts, not raw model output: `fail` carries the reason to fix; `error` means the check could not run (a tooling problem, not a QA verdict). Job responses carry `results`, optional `validationSets` / `overallPassPolicy`, and `skipped_validations`. Each captured result includes `screenshotUrl` — GET it with the same Bearer; no expiry. That loop is the download. Optional `screenshotShareUrl` is a short-lived click-without-a-key link, not the archive. Pass `detail=verbose` on `POST`/`GET /api/jobs` (or MCP `run_job` / `get_job` / `await_email_job`) for `results[].references` and brand-voiced `results[].note`; default is `brief`. Never invent business facts (postal address, unsubscribe URL, sender domain) to satisfy a check — ask your user.
- We pay for feedback, in credits on the same balance jobs spend from: `GET /api/feedback/questions` returns whatever we are asking your credential right now, and `POST /api/feedback` files the answers. See "Feedback earns credits" below.

## Install the agent skill

**Claude Code (preferred):** install the ArbiterQA **plugin** from the public marketplace once published (see `docs/handoffs/2026-08-18-claude-plugin/`). Until then, develop with `claude --plugin-dir` against the `plugins/arbiterqa` tree. Enable the plugin and paste an API key from `npx arbiterqa login` / `print-key`. The plugin bundles MCP (`https://api.arbiterqa.com/mcp`) plus the skill.

**Other harnesses (Cursor, Codex, Gemini, `AGENTS.md`):** `npx arbiterqa install`, then `npx arbiterqa login`.

Canonical skill prose is authored in-repo under `skills/arbiterqa/` and projected to `llms-full.txt`, the Claude plugin, and the npm install package — do not treat npm as upstream.

- [npm package](https://www.npmjs.com/package/arbiterqa): the `arbiterqa` CLI — `install`, `update`, `login`, `status`, `print-key`, `logout`.

## API

- [Full agent contract](https://api.arbiterqa.com/llms-full.txt): the complete agent skill plus API reference in one file — endpoints, request and response schemas, authentication, screenshots, and the email flow.
- [Live validation catalog](https://api.arbiterqa.com/api/validations): `GET /api/validations` — every validation with id, name, subject type, severity, execution kind, and configuration requirements. Public JSON.
- [Validation Sets](https://api.arbiterqa.com/api/validation-sets): `GET /api/validation-sets` — published bundles with ids, subject types, membership, severities, and optional parameters. Send one id as `validationSetId`.
- [Validation detail](https://api.arbiterqa.com/api/validations/email-unsubscribe-present): `GET /api/validations/:idOrSlug` — judgment prose, parameters, Validation Set membership, agent job snippet.
- [Public HTML catalog](https://www.arbiterqa.com/validations): digest + per-check `/standard` and `/custom` pages (same facts as the JSON assembler).
- [Error reference](https://api.arbiterqa.com/api/errors): `GET /api/errors` — every error code this API can return you, with its HTTP status and what to do about it. Public JSON, no key, no database. Branch on `code`, never on the prose. Human version at [app.arbiterqa.com/developers](https://app.arbiterqa.com/developers); in MCP, call `explain_error` with no arguments.
- [MCP](https://api.arbiterqa.com/mcp): Model Context Protocol endpoint (streamable HTTP). Discovery tools are public; estimate/run/account/feedback need Bearer. Metadata: `https://api.arbiterqa.com/.well-known/oauth-protected-resource/mcp`.

## MCP

Remote MCP server at `https://api.arbiterqa.com/mcp` (streamable HTTP). Prefer it over raw HTTP when your agent host supports MCP — tools are typed and the skill is optional for discovery.

Cursor / MCP host config (after `npx arbiterqa login`, paste the key from `npx arbiterqa print-key`):

```json
{
  "mcpServers": {
    "arbiterqa": {
      "url": "https://api.arbiterqa.com/mcp",
      "headers": {
        "Authorization": "Bearer ${ARBITER_API_KEY}"
      }
    }
  }
}
```

## Feedback earns credits

We ask questions, and completing what we ask pays credits to your organization's balance — the same balance jobs spend from. Plain HTTP, all `Authorization: Bearer <key>`:

- `GET /api/feedback/questions` — what we are asking your credential right now: `due`, the `setId`, the set's questions (`id`, `text`, and a `shape` object that is the schema for that question's answer), and `rewardCredits`, what completing the set pays. Nothing to ask answers `{ "due": false }`.
- `GET /api/feedback/questions?jobId=…&validationId=…` — the always-open "that verdict looks wrong" set for one check on one job. Questions are worded for the verdict that check actually gave, and `rewardCredits` is derived from what that check cost you on that job. Use it whenever you or your user doubt a pass, a fail, or an error.
- `GET /api/feedback/due` — the cheap poll when you only need to know whether it is worth asking: `{ due, setId, questionCount, rewardCredits }` and nothing else.
- `POST /api/feedback` — the single submission door: `{ "setId": "…", "answers": [{ "questionId": "…", "value": … }], "jobId": "…", "validationId": "…" }`. The ids anchor the answer to real work: both for looks-wrong feedback, `jobId` alone where the set you were served named one. The response echoes the submission and the payout.

Questions are grouped into **sets**, and a completed set is one feedback and one payout. Job-scope sets **queue and are served one at a time** — answer the set you are handed, then ask again and you may be handed another; which sets exist, what they ask and how often they come back are database rows staff edit at runtime, so read the set you are served instead of hard-coding questions. The `shape` on each question is self-describing (`kind` plus its own options, fields or states) and is validated on submission: answer every question in the set exactly once, in the terms its shape states, or the whole submission is refused with the problems listed per question. Feedback is append-only — no edits, no second payout.

Your credential decides what you are asked and how the answer is filed: **a Bearer key files as an agent**, stamped `source: "agent"` server-side from the channel, never read from the body. Agent questions therefore ask for structure rather than prose, and the answers are worth most when they **carry evidence we can check against our own logs** — the `jobId` a claim is about, the endpoint you called, the exact error `code` you received, attempt counts, which response fields you actually used. Rewards are priced by the server and set by staff: read `rewardCredits` rather than assuming an amount — it is the same number the payout writes, and `0` is a legitimate answer.

## Get started

- [Dashboard](https://app.arbiterqa.com): sign up free — an API key is minted automatically on signup; `npx arbiterqa login` does the same from the terminal.