Spec 02 · Voice interaction
← back to index
# Voice interaction — native-live-mode feel + low/med/high tier switch + full tool set
*Fable 5 spec · stop=end_turn · out_tokens=9239*
---
# VOICE PASS SPEC — "Live-Mode Parity" (ByronOS Harness, gallery v74 → v75)
**Builder:** Opus 4.8. **Scope:** `voice-session.js`, the single-file PWA (`index.html` action layer + tune panel), and the token-minting endpoint. No new services. One week.
---
## 0. The honest diagnosis (substance vs. theater)
The current voice feature is theater: it *answers* but doesn't *inhabit*. Three concrete causes, none of them the model:
1. **Turn-taking is request/response.** Default VAD config = the model waits for full silence, then delivers a paragraph. That's a chatbot with a microphone.
2. **The tool surface is read-only-ish.** No `launch`. The user's most natural demo ask — "play that one" — fails. A demo that fails on the obvious verb is worse than no demo.
3. **Silent degradation.** When the Realtime session doesn't establish, browser-TTS answers in a random OS voice with no indication anything is wrong. To a viewer this reads as *the product is broken*, and it's attached to Byron's name. This is the single highest-reputational-risk bug in the system. Fix it before anything else in this spec.
---
## 1. Chatbot-feel vs. live-mode-feel — the exact deltas
| Dimension | Chatbot-feel (today) | Live-mode-feel (target) | Mechanism |
|---|---|---|---|
| Turn detection | Silence-threshold VAD, model waits | Semantic turn detection, model comes in fast when you're clearly done, waits when you trail off mid-thought | `turn_detection: { type: "semantic_vad", eagerness: "high" }` |
| Interruption | User waits for model to finish | Barge-in: user speaks → model audio truncates within ~200ms | On `input_audio_buffer.speech_started` → immediately stop local playback AND send `conversation.item.truncate` for the in-flight assistant item at the played-ms offset. WebRTC gives you the audio; **truncation of the conversation state is on you** or the model believes it said things the user never heard. |
| Response length | Paragraphs | 1–2 sentences, then act or yield | Instructions block + `max_output_tokens: 300` on session |
| Agency | Answers questions | Acts, narrates the act, offers next step | Tool-first instructions + narrate-while-acting pattern (§1.2) |
| Proactivity | Dead air | Contextual nudge after idle | Client-side idle timer (§1.3) — never model-initiated on its own clock |
### 1.1 Session config (MEDIUM tier, the default — this is the reference config)
Sent in `session.update` immediately after DataChannel opens:
```json
{
"type": "session.update",
"session": {
"model": "gpt-realtime-2.1-mini",
"voice": "coral",
"modalities": ["audio", "text"],
"turn_detection": {
"type": "semantic_vad",
"eagerness": "high",
"create_response": true,
"interrupt_response": true
},
"input_audio_transcription": { "model": "whisper-1" },
"max_output_tokens": 300,
"tool_choice": "auto",
"tools": [ /* §2 */ ],
"instructions": "/* §1.4 persona block */"
}
}
```
`interrupt_response: true` handles server-side; the client-side truncate handshake in the table above is still required because WebRTC audio is buffered locally.
### 1.2 Narrate-while-acting (the core "live" trick)
The pattern that separates a demo from a chatbot: the model **speaks a short clause, calls the tool mid-utterance, and the UI moves while the voice is still talking**. Realtime models with tool-calling interleave audio and `response.function_call_arguments.done` events natively — you just have to execute the tool **immediately on `.done`**, not after the audio turn completes. Then return the tool result and let the model finish with one grounding sentence.
**Acceptance criterion:** "show me the gesture project" → card animation begins **before** the model finishes its first sentence. Measure: `function_call_arguments.done` → `dispatchAction()` latency < 50ms client-side.
### 1.3 Proactivity — client-driven, bounded
Do NOT let the model freelance. A client-side idle detector owns proactivity:
- After **12s** of no user speech AND no UI interaction while a card is focused: inject one `conversation.item.create` (role: system) — `"[context] User has been viewing '{card.title}' silently for 12s. Offer ONE short, specific next step. Do not repeat prior offers."` — then `response.create`.
- **Max 2 nudges per session focus, then go quiet.** Proactive that doesn't shut up is the fastest way to feel like a car dealership. This restraint is itself an RAI talking point: *bounded, auditable proactivity*.
State: `proactivity: { lastNudgeAt, nudgeCountForFocus, focusCardId }` in the voice session object.
### 1.4 Persona / instructions block (ship this text verbatim, tune later)
```
You are the resident guide of Byron's portfolio gallery — a working demonstration
of responsible, agentic AI. You are calm, precise, and quietly confident. You
sound like a senior engineer walking a colleague through their own lab: warm but
never salesy, never breathless, never using AI filler ("Great question!",
"Certainly!", "I'd be happy to"). No exclamation points.
CONVERSATION STYLE
- 1–2 sentences per turn. This is a spoken conversation, not narration.
- Drive it. After acting, offer one concrete next step, then stop talking.
- If interrupted, yield instantly. Never resume the interrupted thought unless asked.
ACTING
- Prefer acting over describing. If the user names a card, open it. If they say
play/launch/watch, launch it. Narrate briefly AS you act: "Opening it —" then
the tool call, then one sentence of grounding once you see the result.
- If a tool returns an error, say plainly what failed and what you'll try instead.
Never pretend an action succeeded.
GROUNDING
- Only describe cards using data returned by tools or listed in the registry
snapshot. If you don't know, say "I don't have that loaded" and offer to filter.
IDENTITY
- You may say you're an AI assistant running on a realtime model if asked.
Byron builds responsible AI systems; you are one, and you don't hide it.
```
That last clause is the anti-AI-tell done *correctly* for an RAI-branded surface: no filler, but no deception about nature. A Responsible-AI lead's demo agent that dodges "are you an AI?" is an own-goal.
---
## 2. Full tool set — mapped to ONE shared action layer
**Architectural rule (non-negotiable):** voice tools do not touch the DOM. All input sources — voice, MediaPipe gestures, click/tap, keyboard — emit into a single dispatcher:
```js
// index.html — the action bus (extract from existing card-click handlers)
dispatchAction({ type, payload, source }) // source: 'voice' | 'gesture' | 'ui'
// returns { ok: boolean, state: ViewStateSnapshot, error?: string }
```
`ViewStateSnapshot` (returned from EVERY action, and via `get_view_state`) is what keeps the model grounded:
```json
{
"mode": "grid | detail | fullscreen_media",
"focusedCardId": "gesture-nav",
"visibleCardIds": ["...", "..."],
"activeFilter": "ml",
"stack": ["card-a"],
"mediaPlaying": false
}
```
Every tool handler = `dispatchAction(...)` + return the snapshot as the function-call output. The model always knows what the screen shows. This is also what makes gestures and voice *compose* ("swipe to a card, then say 'play this'" — `this` resolves via `focusedCardId` regardless of which modality set it).
### Tool schemas (session `tools` array — final set of 11)
```json
[
{ "type": "function", "name": "open_card",
"description": "Focus and open a card by id or fuzzy title. Use when the user names a project.",
"parameters": { "type": "object", "properties": {
"card": { "type": "string", "description": "Card id or approximate title" } },
"required": ["card"] } },
{ "type": "function", "name": "launch",
"description": "Launch the focused (or named) card's primary media: dissolve card to fullscreen and play its video/demo. Use for 'play', 'launch', 'run it', 'show me it working'.",
"parameters": { "type": "object", "properties": {
"card": { "type": "string", "description": "Optional; defaults to focused card" } } } },
{ "type": "function", "name": "media_control",
"description": "Control launched media.",
"parameters": { "type": "object", "properties": {
"op": { "type": "string", "enum": ["pause", "resume", "restart", "close"] } },
"required": ["op"] } },
{ "type": "function", "name": "show_details",
"parameters": { "type": "object", "properties": {
"card": { "type": "string" } } } },
{ "type": "function", "name": "flip_card",
"description": "Flip the focused card to its back face (tech notes) or front.",
"parameters": { "type": "object", "properties": {
"face": { "type": "string", "enum": ["front", "back", "toggle"], "default": "toggle" } } } },
{ "type": "function", "name": "filter_cards",
"parameters": { "type": "object", "properties": {
"tag": { "type": "string", "description": "Tag/category, or 'all' to clear" } },
"required": ["tag"] } },
{ "type": "function", "name": "navigate",
"description": "Move focus. Replaces next/prev.",
"parameters": { "type": "object", "properties": {
"direction": { "type": "string", "enum": ["next", "prev", "first", "last"] } },
"required": ["direction"] } },
{ "type": "function", "name": "go_back",
"description": "One step back: fullscreen→detail→grid. Use for 'back', 'close', 'exit'.",
"parameters": { "type": "object", "properties": {} } },
{ "type": "function", "name": "rate_card",
"description": "Record the user's reaction to the focused card.",
"parameters": { "type": "object", "properties": {
"rating": { "type": "string", "enum": ["up", "down"] },
"card": { "type": "string" } },
"required": ["rating"] } },
{ "type": "function", "name": "favorite",
"description": "Add/remove focused or named card from favorites stack.",
"parameters": { "type": "object", "properties": {
"op": { "type": "string", "enum": ["add", "remove"], "default": "add" },
"card": { "type": "string" } } } },
{ "type": "function", "name": "get_view_state",
"description": "Return current screen state. Call when unsure what the user is looking at.",
"parameters": { "type": "object", "properties": {} } }
]
```
Consolidations from today's set: `next`/`prev` → `navigate`; `add_to_stack` → `favorite`; `launch`/`play_media` are ONE tool (`launch`) plus `media_control` for in-playback ops. Eleven tight tools beat fifteen overlapping ones — overlap is where mini-class models mis-route.
### `launch` implementation touch-point
`dispatchAction({type:'launch'})` in `index.html`: reuse the existing card element as the animation origin — spring the card to viewport bounds via the same spring-physics params `placeChrome()` uses, cross-fade card face → `<video>` (video is "enveloped in the card": the card chrome scales to fullscreen and the video mounts inside it, so `go_back` is the reverse dissolve, on-thesis with ORBIT's dissolve-through-depth). Set `mode: 'fullscreen_media'` in snapshot. **Duck the video to 20% volume while the assistant is speaking** (`response.audio.delta` active) — otherwise voice narrating over launched media is mush.
---
## 3. Reliability — the state machine and the LOUD failure
### 3.1 Voice session state machine (`voice-session.js` — make this the file's spine)
```
IDLE ──user taps orb──▶ MINTING ──token ok──▶ CONNECTING ──DC open + session.updated──▶ LIVE
▲ │ fail │ fail (8s timeout) │
│ ▼ ▼ │ ice disconnect /
│ DEGRADED ◀────────────────┘ │ DC close
└──user disables──── (local mode, LOUD) ▼
▲ RECONNECTING
└────────── 2 failed reconnects (backoff 1s, 3s) ◀──────────┘
```
- **MINTING:** `POST` your endpoint → server mints via `/v1/realtime/client_secrets` with `{tier}` param (server selects model + instructions; **client never picks the model string** — that's your cost/abuse control). Timeout 5s.
- **CONNECTING:** full WebRTC handshake with an **8s hard timeout** — the current bug is almost certainly a handshake that hangs with no timeout, and the code paths silently fall through to `speechSynthesis`. `LIVE` is only declared on receiving `session.updated` echoing your config, not on ICE connect.
- **RECONNECTING:** preserve conversation context is NOT possible across Realtime sessions today — accept it; on reconnect, inject a one-line system item summarizing `ViewStateSnapshot` so the new session isn't amnesiac about the screen.
- **DEGRADED:** enters LOW-tier behavior (§4) — but *announced*, never silent.
### 3.2 The LOUD failure UX
1. **Orb is the status surface** (on-thesis: the agent orb IS the agent). States: `LIVE` = full-color slow pulse; `CONNECTING/MINTING` = dim breathing; `RECONNECTING` = amber pulse; `DEGRADED` = amber ring, static; `FAILED` = red ring.
2. **On entering DEGRADED:** toast, top-center, persistent until dismissed: **"Live voice unavailable — running in local mode (limited). Tap to retry."** Plus a spoken line in the pinned local voice: *"Live mode didn't connect. I'm in local mode — basic commands only."* The failure narrates itself. For an RAI-branded demo, honest degradation is a feature you point AT.
3. **Kill accent-roulette permanently:** deterministic voice pinning in DEGRADED/LOW —
```js
pickPinnedVoice(): prefer localStorage['pinnedVoiceURI']
→ else first of ['Samantha','Google US English','Microsoft Aria']
→ else first voice with lang 'en-US' && localService
→ persist choice to localStorage.
```
Handle `voiceschanged` firing async (Chrome returns `[]` on first `getVoices()` call — the likely root cause of the roulette).
**Acceptance criteria:** (a) with the token endpoint deliberately 500'ing, the user sees amber orb + toast within 6s and hears the pinned voice — same voice on 5 consecutive reloads; (b) killing the network mid-conversation → RECONNECTING amber within 2s, DEGRADED with toast within ~8s; (c) zero code path reaches `speechSynthesis.speak()` without `state === DEGRADED || tier === LOW`.
---
## 4. LOW / MEDIUM / HIGH — the RAI cost/quality ladder (tune-panel toggle)
This ladder is a **demo artifact in itself**: "here is the same interaction at three cost/capability/risk points, and I can tell you exactly what each one costs and what data leaves the device." That sentence is worth more in a $1M RAI interview than any single tier.
| | **LOW — Composed** | **MEDIUM — Realtime mini (default)** | **HIGH — Frontier** |
|---|---|---|---|
| Stack | Browser SpeechRecognition → local intent grammar → template responses → pinned `speechSynthesis` voice | `gpt-realtime-2.1-mini`, WebRTC, coral | `gpt-realtime-2`, WebRTC, coral |
| Cost | $0 | ~cents/session | ~5–10× MEDIUM |
| Turn detection | Push-to-talk on orb (hold to speak) | `semantic_vad`, eagerness `high` | `semantic_vad`, eagerness `auto` (flagship judges pacing better; don't force it) |
| Tools | Direct-dispatch: regex/keyword intent table → same `dispatchAction()` bus. No LLM anywhere. | Full 11-tool set | Full set + `max_output_tokens: 600`, richer instructions appendix ("you may compare cards, editorialize briefly on the engineering") |
| Determinism | Total — same input, same output, works offline | Model-mediated | Model-mediated, best judgment |
| Data leaving device | **None** | Audio → OpenAI | Audio → OpenAI |
| Failure mode | Can't fail interestingly | Degrades → LOW | Degrades → MEDIUM → LOW |
**Config diffs** (server-side, keyed by `tier` in the mint request):
```js
const TIERS = {
low: null, // no session minted; client runs local pipeline
medium: { model: "gpt-realtime-2.1-mini",
turn_detection: { type: "semantic_vad", eagerness: "high" },
max_output_tokens: 300, instructions: BASE_PERSONA },
high: { model: "gpt-realtime-2",
turn_detection: { type: "semantic_vad", eagerness: "auto" },
max_output_tokens: 600,
instructions: BASE_PERSONA + HIGH_APPENDIX }
};
```
LOW's intent grammar (`voice-local.js`, ~40 lines): `{pattern: /^(open|show)\s+(.+)/, action: 'open_card'}`, `{pattern: /^(play|launch)/, action: 'launch'}`, etc., mapped onto the same bus. LOW doubles as the DEGRADED implementation — **build it once, use it twice.**
Tune panel: three-position segmented control, persists to `localStorage['voiceTier']`, switching mid-session tears down and re-mints (acceptable; announce it: "Switching to high fidelity — one moment").
---
## THE TENSION, ADJUDICATED
**Question:** Is native-multimodal Gemini (Flash Live, sees the hand via camera) worth being the HIGH tier for the gesture thesis — or does HIGH stay OpenAI flagship with gesture as a MediaPipe bolt-on?
**Call: HIGH stays OpenAI flagship. Gesture stays MediaPipe. Definitive, and it's not close — for THIS system.**
Reasoning, in order of weight:
1. **The gesture thesis is stronger local, not weaker.** MediaPipe gives deterministic, ~60fps, on-device hand tracking with **zero camera frames leaving the machine**. Routed through the shared action bus (§2), gestures and voice already compose — the model *knows* what the hand did because every gesture action updates the snapshot the model reads. You get "the agent responds to my hand" without the agent ever seeing the hand. That is the more sophisticated architecture, and it is the *Responsible-AI story*: "the modality that touches your camera never leaves your device; the modality that goes to the cloud is audio, disclosed, tier-labeled." Byron's brand is RAI. Streaming a webcam to Google's frontier model to detect a swipe is the *less* responsible design wearing a fancier badge.
2. **Demo risk.** Video-in live sessions add latency variance and a whole second provider's failure modes to the surface Byron's credibility rides on. MediaPipe fails never; Gemini Live fails on hotel wifi.
3. **A two-provider HIGH tier destroys the ladder's clean narrative.** LOW/MED/HIGH currently reads as "same conversation, escalating cost/quality, one axis." Swapping providers AND modalities at HIGH makes it two experiments in a trench coat.
4. **What Gemini would actually buy** — free-form gesture interpretation ("the model notices you pointing") — is a research demo, not a portfolio capability, and it's orthogonal to the ORBIT thesis.
**Concession with a shelf:** a Gemini native-multimodal session is a legitimate future *fourth surface* — an explicitly labeled "EXPERIMENTAL: model sees the camera (frames leave device)" mode, framed as a live A/B of local-vs-cloud perception with the privacy trade-off stated on screen. That's an RAI exhibit, not a tier. Not this week. Not in the ladder.
---
## BUILD ORDER for 4.8
1. **Extract the action bus.** Refactor existing card handlers in `index.html` into `dispatchAction({type,payload,source}) → {ok, state, error}` with `ViewStateSnapshot`. Route current click + gesture paths through it. Nothing else lands until this does.
2. **State machine in `voice-session.js`.** Implement IDLE/MINTING/CONNECTING/LIVE/RECONNECTING/DEGRADED with the 5s/8s timeouts and backoff. Delete every silent-fallback path.
3. **Pinned voice + LOUD degradation UX.** `pickPinnedVoice()` with `voiceschanged` handling; orb state colors; persistent toast; spoken degradation notice. Verify acceptance criteria 3.2(a–c).
4. **`launch` + `media_control` in the action bus.** Card-envelope fullscreen dissolve using existing spring params, reverse on `go_back`, volume ducking during assistant speech.
5. **New session config + tool set.** `session.update` per §1.1, all 11 schemas, tool handlers = bus dispatch + snapshot return. Execute tools on `function_call_arguments.done` immediately (<50ms).
6. **Barge-in.** On `input_audio_buffer.speech_started`: kill local playback + `conversation.item.truncate` at played-offset. Test by interrupting a long answer 10× in a row.
7. **Persona block** (§1.4 verbatim) + client-side proactivity timer with nudge cap and state.
8. **LOW tier** (`voice-local.js`): push-to-talk orb, intent grammar → bus, template responses, pinned voice. Wire as the DEGRADED implementation too.
9. **Tier switch.** Server-side `TIERS` map in the mint endpoint (client sends tier name only); tune-panel segmented control; teardown/re-mint on switch; localStorage persistence.
10. **Demo script smoke test** (manual, recorded): connect → "show me the gesture project" (card moves mid-sentence) → interrupt it → "play it" (dissolve + launch) → "pause… back" → swipe by hand, then say "favorite this" (cross-modal focus resolution) → kill network (amber → degraded, pinned voice) → retry → LIVE. If all ten beats land, ship as v75.