Voice model landscape

← back to index

# Realtime Voice / Speech-to-Speech Landscape — mid-2026
*Captured 2026-07-26 for the benchmarks.arnao.ai conversational-voice row + harness voice decision. Sources verified via web search this date.*

## Interaction modes
1. **Native speech-to-speech (S2S)** — audio in → audio out, one model, no transcribe/reason/synthesize hops. Lowest latency, real barge-in. → the harness voice.
2. **Composed STT→LLM→TTS** — flexible per-stage, but this is the accent-roulette failure mode + added latency. Avoid for live harness.
3. **Voice-quality / cloning layer** — the *produced* voice (Byron's cloned podcast). Separate job.

## Native S2S field

| Provider / model | Released | Price (audio) | Multimodal | Tool calls | Harness note |
|---|---|---|---|---|---|
| **OpenAI gpt-realtime-2.1-mini** | Jul 7 2026 | ~$10/$20 per 1M (~$0.04/min real) | audio+text | ✅ reasoning **+ tool use** in mini now, 25% lower p95 | **Gallery already runs this. Entitlement confirmed.** WebRTC/WS/SIP, barge-in, ephemeral token via `/v1/realtime/client_secrets`. July drop closed the tool-calling gap → this is the default. |
| **OpenAI gpt-realtime-2** | May 7 2026 | $32/$64 per 1M | audio+text | ✅ GPT-5-class reasoning | Flagship. Overkill/expensive for card nav; HIGH-tier candidate. |
| **Google Gemini 3.1 Flash Live** | Mar 24 2026 (preview) | Flash-tier | **audio + video + images** (128K ctx) | ✅ 90.8% ComplexFuncBench, **synchronous only** (won't speak until tool returns) | **Only native model that SEES.** Camera + gesture in one stream → the gesture-thesis fork. Sync tool-calling is the tradeoff. |
| **Amazon Nova 2 Sonic** | Dec 2025 | $3/$12 per 1M (~$0.015/min) | audio | ✅ **async** tool calling, 1M ctx | ~80% cheaper, AWS-native (Bedrock). Async > Gemini sync. The AWS/Kiro brand story — kept as a documented deployment target, NOT a running account (Byron: skip the account). |
| **xAI Grok Voice** (`grok-voice-think-fast-1.0`) | Jun 29 2026 | $0.05/min | audio | ✅ | **Drop-in OpenAI-compatible** — change base URL to `wss://api.x.ai/v1/realtime`, existing voice-session.js works. Zero-migration A/B option. |
| **ElevenLabs ElevenAgents / CAI 2.0** | Jan 2026 | platform | see+hear+files | ✅ + RAG | Sub-100ms, best voice quality, custom turn-taking. Composed platform → reserve for mode ③ (Byron's cloned produced voice). |

## Decisions locked
- **Harness voice = OpenAI gpt-realtime-2.1-mini** (settled; already wired; not the bottleneck).
- **AWS/Nova = documented deployment target only** — one paragraph in the Mentor/Kiro PRD, no new AWS account (Byron's call 2026-07-26). No technical reason to run it; brand story doesn't require running it.
- **Open fork for Fable (spec 02):** does native-multimodal **Gemini** become the HIGH tier for the gesture thesis, or stays OpenAI flagship + gesture as MediaPipe bolt-on.
- **low/med/high switch:** LOW=deterministic composed (pinned voice, cheap, always-works fallback) · MEDIUM=gpt-realtime-2.1-mini · HIGH=frontier (gpt-realtime-2 or Gemini Flash Live).

## Sources
OpenAI voice models (openai.com/index/advancing-voice-intelligence-with-new-models-in-the-api) · gpt-realtime-2.1-mini (marktechpost.com 2026/07/06) · Gemini 3.1 Flash Live (ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview; marktechpost.com 2026/03/26) · Nova 2 Sonic (aws.amazon.com/about-aws/whats-new/2025/12; cloudprice.net/models/amazon-nova-sonic) · Grok Voice (x.ai/news/grok-voice-agent-api) · ElevenLabs 2026 guide (medium.com/the-ai-entrepreneurs).