Spec 06 · Signal curation
← back to index
# Signal curation console + source expansion (Fable 5 spec)
*stop=end_turn · out_tokens=10555*
---
# SIGNAL CURATION + SOURCE EXPANSION — Engineering & Product Spec
**Spec ID:** 04-signal-curation
**Depends on:** 01-harness-architecture (unified owner-token), live quip-service (server.js @127.0.0.1:18801, Tailscale :8443)
**Prime directive:** reuse `update_steering` / `add_context` / `feeds.json`. Anything that writes intent goes through the *same code paths the voice tools already use*. One write path, two front-ends (voice, console).
---
## 0. Ruthless-critic preamble (what's substance, what's theater)
- **Substance today:** two-layer intention model is genuinely good architecture. The atomic-write + backup on `update_steering` is real engineering. The two-way grounded conversation is the differentiator — keep it primary.
- **Theater today:**
1. **The "investment view" is theater.** Finance is dark. A briefing that muses about "AI companies to invest in" with no data behind it is exactly the fabrication pattern a Responsible-AI lead cannot ship under his own name. Either wire real data (below) or the briefing generator must *refuse the segment*. This spec wires it.
2. **Voice-only curation is a demo flex, not a control plane.** You can't audit what you can't see. "I steer it by talking" is great on stage; it's how config rot happens in production. Steering.md has an append marker and no owner ever reads it back — that's write-only memory.
3. **The echo chamber is structural, not incidental.** Byron curated the roster; the roster reflects Byron; Fable narrates the reflection. Fixing this requires a *provenance model and a quota*, not just "more sources" — more sources Byron picks is a bigger mirror.
---
## 1. Component: Owner Console (`/signal/console`)
### 1.1 Posture (this is the tension — full adjudication at the end)
**Conversation remains the only write path for prose intent** (steering appends, context lines, "fewer job advisories"). **The console is the write path only for structured knobs** (weights, keyword chips, advisory-mode toggles, objective status) — things that are miserable by voice ("what are my 43 source weights again?"). The console renders prose layers **read-only** with a literal hint: *"To change this, say it to Signal."* No free-text steering editor in v1. This kills dead-UI risk: every element on the page is either live data or a toggle that works.
### 1.2 Auth
- Accepts **either** the existing cookie-gated key **or** `Authorization: Bearer ${BYRONOS_OWNER_TOKEN}` (spec-01 God-mode token). Implementation: a single `requireOwner(req)` middleware in server.js checking both; existing routes migrate to it opportunistically. No second cookie, no second gate. This also relieves the recurring access pain: the Bearer path works from curl/Shortcuts without the cookie dance.
### 1.3 Files & endpoints
| File | Purpose |
|---|---|
| `public/console.html` | Single static page, no framework, no build step. Fetch + render. |
| `lib/curation.js` | All read/merge/patch logic. Voice tools import the *same functions*. |
| `lib/steering-guard.js` | Leak guard (see 1.5). |
| `signal-data/curation-audit.jsonl` | Append-only audit log. |
**`GET /signal/curation`** (owner) → composite state:
```json
{
"steering": { "content": "…rendered md…", "lastAppend": "2025-06-11T07:14:02Z", "readonly": true },
"personal": {
"contextLines": ["…"],
"objectives": [
{ "id": "obj-rai-roles", "text": "Land principal/distinguished Responsible-AI role",
"keywords": ["responsible ai", "ai governance", "ai policy lead"],
"status": "active", "advisoryMode": "count-only", "created": "…" }
],
"keywords": { "boost": ["frontier eval"], "mute": ["prompt-engineering listicle"] },
"advisories": { "jobs": "count-only", "invest": "full", "papers": "full" }
},
"sources": [
{ "id": "arxiv-cs-ai", "class": "papers", "weight": 1.2, "provenance": "OWN_ROSTER", "lastFetch": "…", "items24h": 14 },
{ "id": "hn-ai-front", "class": "discourse", "weight": 1.0, "provenance": "INDEPENDENT", "items24h": 9 }
],
"xVoices": ["@handle1", "…"],
"audit": [ { "ts": "…", "actor": "voice|console", "op": "set_advisory_mode", "diff": {"jobs": ["full","count-only"]} } ]
}
```
**`PATCH /signal/curation`** (owner) — **structured fields only**; server rejects any attempt to write `steering` or free-text `contextLines` edits (contextLines: add/remove whole lines only, same semantics as `add_context`):
```json
{ "sources": [{"id":"arxiv-cs-ai","weight":0.5}],
"keywords": {"boost":{"add":["mechanistic interp"]}, "mute":{"remove":["…"]}},
"advisories": {"jobs":"count-only"},
"objectives": {"update":[{"id":"obj-rai-roles","status":"paused"}]} }
```
All writes: backup + atomic write (reuse the exact `update_steering` write helper), append to `curation-audit.jsonl` with `actor: "console"`.
**`GET /signal/curation/audit?n=50`** — tail of the audit log. Voice tool calls log here too (`actor: "voice"`, include the transcript snippet that triggered it). *This is the killer feature: Byron can see what he told Signal last Tuesday and what it actually changed.*
### 1.4 New voice tools (registered in existing tool dispatch)
Thin wrappers over `lib/curation.js` — same functions the PATCH endpoint calls:
- `set_advisory_mode(topic, mode)` — mode ∈ `full | count-only | off`. This directly implements *"fewer guide-point job advisories right now, just remind me N came up."*
- `add_objective(text, keywords[])` / `update_objective(id, {status|keywords|advisoryMode})`
- `boost_keyword(term)` / `mute_keyword(term)`
- `set_source_weight(sourceId, weight)` — fuzzy-matches source names ("turn arxiv down").
### 1.5 Steering leak guard (`steering-guard.js`)
`update_steering` is shared DNA consumed by the **enterprise RAI Report**. Personal job-hunt content leaking there is a career-embarrassment bug, not a nit.
- **Layer 1 (regex, synchronous, free):** blocklist of personal markers — role-hunt phrases ("my job search", "roles for me", salary figures, "Byron should apply"), calendar/email personal refs. Match → hard reject.
- **Layer 2 (LLM classifier, one cheap call):** classify the proposed append `SHARED | PERSONAL | AMBIGUOUS`. `PERSONAL` → **auto-reroute** to `add_context` (personal layer) and tell Byron aloud: *"That's personal — I put it in your Signal context, not the shared house DNA."* `AMBIGUOUS` → ask one clarifying question in-conversation.
- Guard applies identically to any future console steering path. Guard decisions logged to audit.
**Acceptance criteria (console):**
- [ ] Console loads behind owner auth (both cookie and Bearer paths tested).
- [ ] Every weight/toggle change round-trips: PATCH → settings file → next briefing generation reflects it → audit entry exists.
- [ ] Voice `set_advisory_mode("jobs","count-only")` and console toggle produce byte-identical settings diffs and distinguishable audit entries.
- [ ] `update_steering` with "remind me about principal RAI openings" is rerouted to personal context, never lands in steering.md; RAI Report generated afterward contains zero personal markers.
- [ ] Zero dead UI: no button on the page that doesn't hit a working endpoint.
---
## 2. Component: Source expansion — breaking the echo chamber
### 2.0 The structural fix: provenance + quota
Every ingested item gets `provenance: OWN_ROSTER | INDEPENDENT` and `confidence: FILED | API | REPORTED | UNVERIFIED`. The briefing composer enforces: **≥30% of narrative items per briefing must be INDEPENDENT**, and a mandatory **dissent slot** — if any independent source contradicts the roster consensus on a topic, Fable must voice it ("Your feeds say X; Reuters and the HN thread say Y"). Without the quota, adding sources just widens the mirror.
### 2.1 Jobs class (`collectors/jobs-collector.py` → `signal-data/jobs/{date}.json`)
**Honest data paths, ranked:**
1. **LinkedIn — the honest answer is: not via API, not via scraping.** No public job-search API; scraping violates ToS, is aggressively fingerprinted, and "Responsible-AI lead caught scraping LinkedIn" is a self-inflicted headline. **The honest path: LinkedIn Job Alert emails.** Byron configures saved searches ("Principal Responsible AI", "Director AI Governance") → alert emails arrive → the pipeline *already ingests email* → `jobs-email-parser` extracts title/company/link. Zero ToS risk, and it's LinkedIn's own relevance ranking working for free.
2. **Indeed:** publisher API is closed; scraping is ToS-risky. **Skip Indeed entirely.** Replace with:
3. **Greenhouse/Lever/Ashby public board JSON** — fully public, documented, legal. Maintain a `companies.json` watchlist (Anthropic, OpenAI, DeepMind, Scale, labs + hyperscaler competitors); poll `boards-api.greenhouse.io/v1/boards/{co}/jobs` etc., filter by objective keywords. This is the *highest-signal* source: postings direct from source, hours-fresh.
4. **Adzuna API** (free tier, real REST API, includes salary estimates *labeled as estimates*) — broad-market role-title trend signal: "what big titles is the market actually posting."
5. **HN "Who's Hiring"** monthly via Algolia API — leading-indicator for what startups want.
**Wrongness budget:** every surfaced posting must have a live URL HEAD-checked within 24h, else labeled `UNVERIFIED` and excluded from the spoken brief. Salary never stated unless present in the posting; Adzuna estimates spoken as "Adzuna estimates." Dedupe by `hash(company+title-normalized)`.
**Fable prompt contract:** jobs items arrive as `{title, company, url, source, posted, confidence, matchedObjective}`. Rendering obeys `advisories.jobs`: `full` → narrative treatment; `count-only` → *"Seven principal-level RAI roles surfaced today, three at frontier labs — say 'jobs' if you want them"* (and `/signal/ask` can expand, because the digest is in the grounding context); `off` → silence, digest still written to disk.
### 2.2 Market/finance class (`collectors/market-collector.py` → `signal-data/market/{date}.json`)
Finance is currently dark and the briefing must not pretend otherwise. Real sources, ranked by authority:
1. **SEC EDGAR full-text search + filings feed** (free, no key, authoritative). Watch **S-1 / S-1A filings** (the actual IPO signal, weeks before press coverage) filtered by AI-relevant SIC codes + keyword match in filing text. Confidence tier: `FILED` — the only tier where Fable may state facts declaratively.
2. **Finnhub free tier** — IPO calendar, company news endpoints. Tier: `API`.
3. **yfinance** for public AI-comp basket (NVDA, MSFT, GOOGL, AI-adjacent) — daily moves as context, not analysis.
4. **Private-market funding rounds: the honest answer is there is no good free source.** Crunchbase is $$$$. Fallback: LLM extraction from ingested news, always labeled `REPORTED` with attribution: *"The Information reports — unconfirmed — that…"* Never a bare number.
**RAI posture (non-negotiable):** the market segment carries a spoken frame — *"market observation, not investment advice"* — once per briefing. Every figure cites source + timestamp. If collectors return empty, the generator **says the segment is dark**, it does not improvise. This posture is itself brand-relevant: a personal briefing tool that visibly refuses to fabricate is a $1M-role artifact.
### 2.3 Independent discourse class (`collectors/discourse-collector.py`)
Sources *not chosen item-by-item by Byron* — chosen by mechanism:
- **HN front page AI stories** (Algolia API) — what engineers-at-large are reading, not who Byron follows.
- **Google News RSS** queries generated from *objective keywords* (RSS consumption is fine) — mainstream/press counterweight to X-VOICES.
- **arXiv adjacent-category sampler** — deliberately samples 2–3 papers/day from categories *near but outside* his keyword set. Anti-filter-bubble by construction.
- **Reddit JSON** (r/MachineLearning top, polite rate limits) — practitioner dissent channel.
All tagged `INDEPENDENT`. These feed the quota and the dissent slot.
**Acceptance criteria (sources):**
- [ ] `jobs-collector` produces a daily digest with ≥1 Greenhouse-direct posting when watchlist companies have live RAI-adjacent postings; every item URL verified.
- [ ] An S-1 filed by a watchlist-relevant AI company appears in the market digest within 24h with `confidence: FILED` and EDGAR link.
- [ ] A generated briefing with market data dark contains the explicit "market segment is dark today" line and zero invented figures (spot-check 5 briefings).
- [ ] Briefing composer rejects composition if <30% INDEPENDENT items available and instead flags the shortfall aloud.
---
## 3. Component: Objective → Research loop (`objective-research.py`, nightly via launchd)
For each `status: active` objective:
1. **Query expansion:** objective keywords → one LLM call producing synonyms + adjacent titles ("Responsible AI" → "AI governance", "trust & safety", "AI assurance", "model policy"). Cached per objective; regenerated on keyword edit.
2. **Fan-out by objective type:** `obj-rai-roles` → jobs collectors + discourse queries ("who's hiring heads of AI governance"); `obj-ai-invest` → EDGAR + Finnhub + news extraction. Mapping declared in the objective record (`sourceClasses: ["jobs","discourse"]`), editable in console.
3. **Scoring:** `relevance × freshness × independenceBonus(1.3 for INDEPENDENT)`, with a hard cap: **≤40% OWN_ROSTER items per objective digest.** This is the anti-echo trap mechanism — the loop *cannot* satisfy an objective purely from Byron's own mirror.
4. **Tie-back:** one generated "so-what" line per digest connecting findings to Byron's current work — generated *only* from cited items + personal contextLines, never from model priors ("Two of today's principal postings list ISO 42001 experience — you shipped that assessment in April").
5. **Output:** `signal-data/objectives/{date}-{objId}.json` → consumed by the briefing generator, rendered per `advisoryMode`, fully available to `/signal/ask` grounding so Byron can drill in by voice.
**Acceptance criteria:**
- [ ] Adding an objective by voice ("track principal RAI roles for me") creates the record, and the next nightly run produces a digest.
- [ ] No digest exceeds the 40% OWN_ROSTER cap; if independent sources are empty, digest says so rather than backfilling from roster.
- [ ] Every tie-back line's claims trace to a cited item (audit sample of 10).
---
## 4. Functioning-TODAY slice
Deployable onto live quip-service today, no collectors required:
1. **Settings schema extension** (`objectives`, `keywords`, `advisories`) — additive, backward-compatible, default `advisories.jobs: "full"`.
2. **`lib/curation.js`** with atomic-write helper reuse + `curation-audit.jsonl`.
3. **Four voice tools** (`set_advisory_mode`, `add_objective`, `boost_keyword`/`mute_keyword`, `set_source_weight`) registered in the existing tool dispatch. Byron can *today* say "cut the job advisories to a count" and "add an objective: principal RAI roles" and the next briefing obeys.
4. **Briefing generator honors `advisoryMode`** — count-only rendering wired into the existing prompt assembly; keywords/objectives injected as boost/mute hints into the existing item-selection stage (works against current feeds even before new collectors exist).
5. **`GET /signal/curation` + read-only `console.html`** behind `requireOwner` (cookie + Bearer). View steering, context, objectives, weights, audit. Zero write UI today.
**Today's acceptance test:** Byron says "fewer job advisories, just give me a count" → tool fires → audit line written → evening briefing renders count-only → console shows the change and who made it. If that whole loop works, ship.
**Staged:** PATCH endpoint + console toggles (week 1); steering leak-guard classifier (week 1 — before any heavy steering use); jobs collector incl. LinkedIn-alert email parser (week 1–2); market collector EDGAR+Finnhub (week 2); discourse collector + provenance quota + dissent slot (week 2–3); objective-research nightly loop (week 3).
---
## (a) THE TENSION — adjudicated
**Should the console be a new keyed surface, or does everything stay conversation-first with a thin read-only view?**
**Call: build the console page — but split the write authority by data type, permanently.** Conversation-first is the *only* write path for prose intent (steering, context, "how I want to be briefed"), because voice is where Byron's editorial judgment actually lives and duplicating a free-text steering editor creates two divergent write paths and a dead textarea. But **structured state — 40+ source weights, keyword sets, objective status, advisory modes — is unauditable and unmanageable by voice alone**; pretending otherwise is demo-thinking. The console is one static HTML page, read-first, with toggles/chips *only* for structured knobs, and every toggle calls the exact same `lib/curation.js` functions the voice tools call. The non-negotiable component is the **audit log**: "conversation-first" without a visible record of what your conversations changed is write-only configuration — that's the actual bug in the current system, and a thin read-only view without editable structured knobs would fix the visibility half while leaving Byron voice-navigating a weights table, which nobody will ever do twice. Low complexity is preserved: one page, no framework, no second write path, nothing on it that doesn't work.
## (b) BUILD ORDER for 4.8
1. **[TODAY]** Extend `signal-settings.json` schema (objectives/keywords/advisories) + `lib/curation.js` (atomic writes, `curation-audit.jsonl`).
2. **[TODAY]** Register voice tools `set_advisory_mode`, `add_objective`/`update_objective`, `boost_keyword`/`mute_keyword`, `set_source_weight` in existing tool dispatch; audit-log all existing `update_steering`/`add_context` calls too.
3. **[TODAY]** Wire `advisoryMode` + keyword boosts/mutes into the briefing generator's item selection and prompt assembly.
4. **[TODAY]** `requireOwner` middleware (cookie OR spec-01 Bearer token); `GET /signal/curation` + `GET /signal/curation/audit`; read-only `public/console.html`.
5. **[STAGED — wk 1]** `PATCH /signal/curation` (structured-only) + console toggles/chips; reject prose writes at the server.
6. **[STAGED — wk 1]** `steering-guard.js` (regex layer + LLM classifier + auto-reroute to `add_context`); apply to voice tool. Ship before Byron leans harder on steering.
7. **[STAGED — wk 1–2]** `jobs-collector.py`: LinkedIn-alert email parser (reuses email ingest) + Greenhouse/Lever/Ashby watchlist + Adzuna + HN Who's Hiring; URL verification; digest → briefing per advisoryMode.
8. **[STAGED — wk 2]** `market-collector.py`: EDGAR S-1 watch + Finnhub IPO calendar + yfinance basket; confidence tiers; "dark segment" refusal path in generator.
9. **[STAGED — wk 2–3]** `discourse-collector.py` + provenance tagging across all sources + 30% INDEPENDENT quota + dissent slot in composer.
10. **[STAGED — wk 3]** `objective-research.py` nightly: query expansion, fan-out, 40% OWN_ROSTER cap, cited tie-back lines, digests into `/signal/ask` grounding.
Steps 1–4 are a functioning-today slice: Byron curates by voice, controls advisory volume, and *sees* his own intention layers for the first time — tonight's briefing already behaves differently. Everything after that is breaking the mirror.