`README.md`

101 lines
# Model Arena

Fan ONE real-world build challenge out to multiple AI models, render each
model's single-file HTML artifact side-by-side in sandboxed iframes, crown a
winner, and build a **real-world win-rate ledger** per model over time.

Seeded by Julian Goldie SEO's Jul 20 2026 X post (Kimi K3 vs Claude Fable 5 vs
GPT 5.6 head-to-head): *"Don't marry one AI model. Build a workflow where every
model does the job it performs best at — trust real-world testing over
leaderboard screenshots."* See `BRIEF.md`.

## Run

```sh
node server.js            # http://localhost:9758, basic auth admin / DW2024!
PORT=4000 node server.js  # custom port
BASIC_AUTH= node server.js       # disable the gate (local only)
BASIC_AUTH=u:p node server.js    # custom credentials
```

Zero dependencies. State lives in `data/` (challenges.json, artifacts/,
votes.jsonl, costlog.jsonl).

## Models

- **Local ($0, pre-checked):** qwen3:14b · gemma3:12b · hermes3:8b (Mac2
  Ollama) + qwen2.5:7b (Mac1 Ollama at 192.168.1.133). Generations are
  serialized per host so Ollama doesn't thrash model loads; hosts run in
  parallel.
- **Metered (opt-in per battle, never pre-checked):** Claude Fable 5, Kimi
  K2.5, GPT-5.3, Grok 4.5. Each needs its env key (`ANTHROPIC_API_KEY`,
  `MOONSHOT_API_KEY`, `OPENAI_API_KEY`, `XAI_API_KEY` — source from
  secrets-manager). Estimated cost shows in the UI before the run; actual
  token cost is appended to `data/costlog.jsonl` AND `~/.claude/cost-ledger.jsonl`.

## API

- `GET  /api/models` — roster + availability + est cost
- `GET  /api/challenges?sort=newest|oldest|title|models|winner`
- `POST /api/challenges {title, prompt, models[]}` — starts the battle
- `GET  /api/challenges/:id` · `DELETE /api/challenges/:id`
- `POST /api/challenges/:id/vote {winner}` — crown, appends to votes.jsonl
- `POST /api/challenges/:id/retry/:model`
- `GET  /api/ledger` — per-model battles/wins/win-rate/errors/avg-time/spend
- `GET  /artifact/:id/:model` — the raw artifact, served with a locked-down
  CSP (no network); embedded with `sandbox="allow-scripts"`

## Safety rails

- Artifacts are untrusted model output: sandboxed iframes + CSP `default-src 'none'`.
- Basic auth on by default.
- $0 by default; paid calls are per-battle opt-in with cost surfaced + ledgered.
- No deploy config on purpose — launch is Steve-gated per ACCELERATOR.md.

## Judging & analytics (added via /yolo run 2026-07-22)

- **Artifact thumbnails** — every finished artifact is headless-rendered to a PNG (`/thumb/:id/:model`); the challenge grid shows a strip of real renders and battle panes show a click-to-run-live poster.
- **AI auto-referee** — a local vision model (`qwen2.5vl:7b`, `VISION_MODEL` to override) *looks at* each rendered artifact and scores it 0–10 with a one-line reason; auto-runs when a battle settles and surfaces an **AI PICK** alongside the human 👑 crown. `$0` local. `POST /api/challenges/:id/judge` to re-run.
- **ELO leaderboard** — ranked by ELO (K=24, crowned model beats every other that rendered) plus win%, cost-per-win, avg AI score, and a **human/AI agreement %** meta-metric.
- **Blind judging** — toggle to anonymize models to "Model A/B/C" until a winner is crowned (kills brand bias).
- **Real-workload presets** — a 🏭 Real Work challenge set (luxury product page, sortable catalog, marketing email, room visualizer, JSON→grid) beside the 🎮 Games set, moving the arena toward Steve's actual production decisions.
- **Remix mode** — `POST /api/challenges/:id/remix` feeds the best artifact's HTML back to the models as "improve on this," spawning an evolutionary round (`remixOf`/`remixSource` lineage).
- **Source view** — `</>` per pane shows the artifact's raw HTML.
- **Daily generator** — `scripts/daily-challenge.js` runs a `$0` all-local battle on a rotating prompt + AI-judges it, to accrue real data. launchd install is drafted (gated) in `~/.claude/yolo-queue/pending-approval/`.

### More analytics (yolo iteration 2)
- **Per-category leaderboards** — challenges auto-tag Games / Real Work / Custom; leaderboard filters ELO by category (`/api/ledger?category=`).
- **Head-to-head matrix** — pairwise W–L (row vs column) from crowned battles, under the leaderboard.
- **Model profile page** — click a leaderboard row for that model's full artifact gallery + aggregate stats (`/api/models/:id/profile`).
- **Export battle** — `/export/:id` downloads a self-contained standalone HTML (artifacts inlined via sandboxed `srcdoc`), renders offline.
- **Consensus referee** — the AI referee is now a *panel* of local vision models (`VISION_MODELS`, default `qwen2.5vl:7b,minicpm-v`) with a spread/disagreement flag.

## Design tools (🎨 per-battle toggle)

Models can be given a graphic-design tool belt (`design-tools.js`):

- **opendesign** — curated palettes (by mood), typography stacks + modular scale,
  layout/spacing recipes, elevation shadows, inline SVG icon paths.
- **hyperframes** — motion catalog: named easings, entrance keyframes, stagger /
  text-effect / ambient-background snippets (pure CSS/vanilla JS — artifacts run
  under CSP `default-src 'none'`, so no CDN libraries).
- **figma** — live brand tokens (most-used fills + named styles) from a Figma
  file via REST. Needs `FIGMA_TOKEN` (optional `FIGMA_ARENA_FILE` default file);
  degrades gracefully when unset.

Tool-capable models (GPT, Grok, Kimi, Qwen3, Hermes3) get a real function-calling
loop (≤6 rounds, all tokens cost-logged as `model-arena-tools`); models without
tool support (Gemma3, claude-code CLI) get the same essentials injected as a
compact design pack. Each run records `toolCalls`, badged 🎨 in the battle view.

### photoshop tool (Adobe Firefly Services)

`photoshop` joins the belt when any image backend is configured: Adobe Firefly (preferred) via `ADOBE_CLIENT_ID` +
`ADOBE_CLIENT_SECRET` are in `.env` (routed via secrets-manager):
`generate_image` (Firefly text-to-image → jpeg asset) and `remove_background`
(Photoshop cutout — Adobe fetches the asset from the auth-exempt
`/ps-asset/…` URL and PUTs the result to a single-use `/ps-asset-put/…`
token). Models inline results via `{{PS_ASSET:id}}` placeholders, swapped
for data URIs at artifact save (CSP already allows `img-src data:`).
~$0.03-0.04/op, logged to both cost ledgers.