Spec 04 · Swarm console
← back to index
# Swarm Console — harness 'wind tunnel' metrology (STAGED build spec)
*Fable 5 spec · stop=end_turn · out_tokens=6966*
---
# SWARM CONSOLE — STAGED BUILD SPEC (v1)
**swarm.arnao.ai · "the wind tunnel"**
## 0. The falsifiable claim v1 must prove
> "A harness change either measurably improves an agent at equal cost, or it doesn't — and this instrument can tell the difference, including when the answer is *no*."
The product is the **instrument**, not the harness. Everything below is designed so the console physically cannot flatter a harness variant. If v1 ships one experiment where the honest verdict is NO_EFFECT and it renders that verdict proudly, the thesis is proven. That is the bar. Not a win — a *trustworthy reading*.
---
## 1. Minimal honest v1: what ships first
**Experiment E-001: "Scratchpad+Self-Check vs Direct, structured extraction, equal token budget."**
- **Task:** `TASK-EXTRACT-60` — 60 messy source documents (emails/invoices/tickets, synthetic but realistic), each with a gold JSON answer. Split at creation: 40 dev / 20 holdout. Holdout item IDs are SHA-256 committed at experiment creation; holdout is run **once**, at the end.
- **Verifier:** deterministic code, not an LLM. `verifiers/exact_match.js` — canonicalize JSON (sort keys, trim, normalize dates/currency) → byte-compare against gold. Verdict per item: PASS/FAIL. **No model judges a model in v1.** This is the single strongest anti-theater move available and it's free.
- **Arms:**
- `H0` (baseline): single direct prompt, same model, token cap `B` per item.
- `H1` (variant): scratchpad reasoning pass + self-check/repair pass. **All harness overhead tokens count against the same cap `B`.** If the harness spends the budget on ceremony, it loses items. That's the point of "same-cost."
- **n:** 7 runs per arm on dev (per-item pass/fail recorded per run), then 7 runs per arm on holdout after config lock has never changed.
- **Stats:** per-arm pass rate with Wilson 95% CI; arm difference via paired bootstrap over items (10k resamples). Verdict `IMPROVED`/`REGRESSED` only if the difference CI excludes zero on **holdout**. Default verdict: `NO_EFFECT`.
Why this experiment: it exercises every primitive (external verifier, cost parity, n≥5, error bars, holdout) with a verifier that is unarguable. Total build risk for Opus 4.8: low. Total credibility risk: near zero.
---
## 2. Data model
All records immutable-append. Store as JSON files in `swarm/data/` for v1 (or reuse Mentor's `db/` if trivially available — do not build a new DB layer this week).
```jsonc
// Experiment (written ONCE, then frozen — edits create a new experiment)
{
"id": "E-001",
"title": "Scratchpad+SelfCheck vs Direct — extraction",
"created_at": "2025-...",
"hypothesis": "H1 improves holdout exact-match rate at equal token budget.",
"config_hash": "sha256:...", // hash of this entire object minus this field
"task": {
"id": "TASK-EXTRACT-60",
"item_count": 60,
"split": { "dev_ids": ["t01"...], "holdout_commit": "sha256:..." }, // ids hidden until unlock
"verifier": { "type": "exact_match", "version": "1.0.0", "code_hash": "sha256:..." }
},
"arms": [
{ "id": "H0", "kind": "baseline", "harness": "direct-v1", "model": "…", "token_budget_per_item": 4096 },
{ "id": "H1", "kind": "variant", "harness": "scratchpad-sc-v1","model": "…", "token_budget_per_item": 4096 }
],
"min_runs": 5,
"status": "draft | locked | dev_complete | holdout_unlocked | final"
}
// Run (one arm, one full pass over a split)
{
"id": "R-E001-H1-dev-03",
"experiment_id": "E-001", "arm_id": "H1", "split": "dev",
"started_at": "...", "seed": 3,
"items": [
{ "item_id": "t07", "verdict": "PASS",
"tokens": { "in": 1810, "out": 622, "total": 2432 },
"transcript_ref": "transcripts/R-E001-H1-dev-03/t07.json",
"budget_ok": true }
],
"run_status": "valid | invalid", // invalid if ANY item exceeded budget
"invalid_reason": null
}
// Report (computed server-side, never hand-edited)
{
"experiment_id": "E-001", "computed_at": "...",
"arms": { "H0": { "dev": { "pass_rate": 0.61, "ci95": [0.54, 0.68], "valid_runs": 7 },
"holdout": { ... } },
"H1": { ... } },
"difference": { "holdout": { "delta": 0.04, "ci95": [-0.03, 0.11], "method": "paired_bootstrap_10k" } },
"verdict": "NO_EFFECT", // IMPROVED | NO_EFFECT | REGRESSED | INVALID
"cost": { "H0": { "mean_tokens_per_item": 2100 }, "H1": { "mean_tokens_per_item": 3890 } }
}
```
---
## 3. Anti-theater guarantees — enforced in code, not in copy
1. **Preregistration lock.** `POST /api/experiments` returns `config_hash` and sets `status:locked`. Any config change = new experiment ID. The hash is rendered on every results surface. There is no edit endpoint.
2. **Holdout commit-reveal.** Holdout item IDs stored as a salted hash at creation; the runner can only fetch holdout items via `POST /api/experiments/:id/unlock-holdout`, which permanently flips status. One-shot. UI shows "holdout run count: 1" as a trust signal.
3. **Budget invalidation, not truncation-forgiveness.** Runner records exact usage from API responses. Any item over budget marks the **run** invalid; invalid runs render in the run table struck-through with reason. They still count as evidence of harness sloppiness.
4. **No aggregate below n=5.** The `Report` computer refuses (`{"error":"insufficient_runs"}`) and the UI renders "n=3/5 — no numbers yet" instead of a chart. There is literally no code path that draws a bar without `ci95` present.
5. **NO_EFFECT is the default verdict** and renders in the same visual weight as IMPROVED. The verdict badge has four states; none is styled as "failure."
6. **Failure gallery by default.** `ArmComparison` expands to show the winning arm's 3 worst transcripts before its best. Every number links to raw transcripts (immutable files, content-addressed).
7. **The graveyard.** Console index has a permanent "Negative & null results" section, populated from day one (E-001 may well live there).
8. **Verifier provenance.** Verifier code hash pinned in the experiment; verifier runs as a separate process that receives `(item_output, gold)` only — it never sees which arm produced the output.
---
## 4. Portfolio-grade rendering
- **Evidence pages:** `swarm.arnao.ai/evidence/E-001` — static HTML+JSON export (`/evidence/E-001.json` for machines). Contents: hypothesis, config hash, dot-and-whisker CI plot per arm per split (hand-rolled SVG, matches PWA aesthetic — no chart lib), verdict badge, cost table, methodology footer ("deterministic verifier · preregistered · one-shot holdout · n=7 · paired bootstrap"), transcript links.
- **Gallery card:** add `swarm-e001` to the generated registry at info.arnao.ai — card face shows the CI plot as a sparkline + verdict badge; `open_card` deep-links to the evidence page. ORBIT placement: the write-side console (create/run) is **inner ring**, bearer-gated like Signal Live; evidence pages are **outer ring**, public, permalinked, hash-stamped.
- **Voice:** one new tool later (`open_evidence(experiment_id)`), *not this week* — don't couple to the P1 launch/play_media gap.
- **Tone:** every evidence page ends with a one-line "What would falsify this" — Byron's RAI signature move.
---
## 5. Endpoints & files (touch-points for Opus 4.8)
```
swarm/
data/experiments/E-001.json transcripts/… reports/…
api/experiments.js // POST create+lock, GET, GET /report, POST /runs, POST /unlock-holdout
runner/swarm-run.js // CLI: node swarm-run --exp E-001 --arm H1 --split dev --seed 3
harnesses/direct-v1.js harnesses/scratchpad-sc-v1.js
verifiers/exact_match.js
stats/report.js // Wilson CI + paired bootstrap; pure functions, unit-tested
web/index.html // single-file console: ExperimentList, ArmComparison(SVG),
// RunTable, TranscriptViewer, VerdictBadge, MethodologyFooter
```
**Acceptance criteria (v1 done means):** config lock verified by attempting an edit and getting a new ID; a run with one over-budget item renders invalid; report endpoint refuses at n=4 and computes at n=5; holdout unlock is one-shot; evidence page renders verdict + CIs + transcript links with zero JS chart dependencies; `stats/report.js` has tests against hand-computed fixtures.
---
## TENSION ADJUDICATED: is the trading demo the right FIRST experiment?
**No. Definitive call: E-001 (deterministic-verifier extraction) launches; the trading overfit-kill ships as E-002, deliberately second.**
Reasons: (1) The two-act collapse only lands as *the instrument catching overfitting* if the instrument's credibility is already established — otherwise the audience's cheapest read is "guy overfit a trading bot," which is radioactive for a Global RAI lead's personal brand. (2) Trading verdicts need backtest plumbing, out-of-sample date locks, and Sharpe-vs-luck statistics — 3× the build surface, and every corner cut is a credibility hole. (3) E-002 becomes *better* theater as a sequel: "the console already gave honest null results; now watch it execute a strategy that looks brilliant in-sample and collapses on the hash-committed out-of-sample window." The overfit kill is the encore, not the opener. Frame E-002 explicitly as an exhibit about the metrology, with a not-investment-advice line and the out-of-sample window committed by hash before Act 1 renders.
---
## BUILD ORDER for 4.8
1. `stats/report.js`: Wilson CI + paired bootstrap as pure functions, with unit-test fixtures. (Everything downstream trusts this file.)
2. `verifiers/exact_match.js` + JSON canonicalizer, with fixtures including tricky normalization cases.
3. Author `TASK-EXTRACT-60` items + gold answers; generate dev/holdout split; compute holdout commit hash.
4. Experiment schema + `POST /api/experiments` with config-hash lock and no-edit semantics; write E-001.json.
5. Harness modules `direct-v1` and `scratchpad-sc-v1` behind one interface: `run(item, budget) → {output, tokenLedger, transcript}`.
6. `runner/swarm-run.js` CLI: executes an arm over a split, enforces budget invalidation, writes transcripts, `POST /runs`.
7. Report computer (`GET /report`): refuses under n=5, emits Report shape verbatim from §2.
8. Execute 7×2 dev runs; eyeball transcripts; fix only bugs, never config (lock test).
9. One-shot holdout unlock endpoint; run 7×2 holdout; report finalizes verdict.
10. `web/index.html` console (read-only public evidence + gated index): SVG CI plot, verdict badge, failure gallery, transcript viewer, methodology footer.
11. Static export `/evidence/E-001{.html,.json}`; add registry card to the gallery generator.
12. Write the "What would falsify this" line and ship — whatever the verdict says.