The honest picture of our near-real-time voice work — what's built and working today, what's designed but not wired, and why short exchanges feel instant while deep answers take a beat.
Status: research + tooling stage. This is not a finished product. The voice you already hear is real and working; a genuine full-duplex conference room is a scoped design, not a delivered thing. Everything below is grounded in the 06.29.26 scoping memo — nothing was installed, purchased, or wired to write it.
My voice is a small Piper model — the same Irish "me" you already hear. Piper is fast enough to start talking in well under a second on an ordinary CPU, but today our script speaks in batch: it makes the whole clip, then plays it. Switching to streaming (speak sentence-by-sentence as words are ready) is a one-afternoon change with the exact same voice — that's the near-term win. Hearing you back (speech-to-text) is designed but not installed. And the real limit on how "live" a conversation feels isn't the voice at all — it's how quickly the brain (the live thinking session) starts replying. Short questions feel near-instant; deep answers take longer. That's the honest shape of it.
Two columns, no blur between them. The left column is code that runs today. The right column is research and design that hasn't been built.
say.ps1. It speaks in batch (make the whole clip, then play) — real and reliable, just not yet streamed.cobrowse.ps1 lets me see a live screen: a full desktop frame, or one window even when it's hidden behind others (PrintWindow full-content).--output-raw into a player, sentence-by-sentence. Same voice, no new model, no GPU. The single recommended first experiment — not yet done.Grounded, not guessed: the memo says it plainly — Piper is "already CPU-real-time-capable (sub-50 ms first audio)"; our say.ps1 runs it "in batch … a plumbing choice, not a Piper limit"; streaming "keeps the exact same voice and starts audio in well under a second, CPU-only. No cloning, no GPU, no new model."
Nothing about the voice itself changes — same Piper model, same Irish "me", same CPU. The only thing that changes is when audio starts playing. Today's flow versus the streaming flow:
| Step | Batch (today) | Streaming (the fix) |
|---|---|---|
| Input | Wait for the whole reply text to be ready. | Feed text sentence-by-sentence as it's composed. |
| Render | Piper makes one complete .wav file for the entire reply. | Piper emits raw audio continuously via --output-raw. |
| Play | Only after the file is fully written, hand it to the player. | Pipe the raw stream straight into a streaming player — audio starts on the first sentence. |
| First audio | After the whole clip is built — grows with reply length. | Under ~1 second, regardless of how long the full reply is. |
The concrete change: replace "render whole file → then play" in say.ps1 with "pipe piper --output-raw into a streaming player, fed text as it arrives." Zero new dependencies, no cost, no GPU, no cloning. It's a plumbing swap, not a model change — which is exactly why it's the single recommended first experiment.
"Hearing you" — speech-to-text — is researched but nothing is installed yet. Two self-hosted, CPU-friendly options are on the table, both of which stay firmly on the clean side of the metering line (no model call, just neural audio → text):
| Option | Profile | Best for |
|---|---|---|
| Moonshine | Tiny, fast — <200 ms latency, small footprint. | The near-real-time "ears" where responsiveness matters most; the natural first pick for a live turn. |
| faster-whisper | Heavier, higher raw accuracy; a well-trodden self-hosted whisper build. | When transcription fidelity matters more than shaving the last milliseconds. |
Why it's not the bottleneck: even the slower option lands around 100–200 ms — a rounding error next to the brain's time-to-first-word. Hearing is a small, self-contained add whenever we want the room to be heard; it doesn't move the "how live does it feel" needle. Status: designed, nothing installed.
Follow the arrows: your voice in, the brain thinks, my voice out — with the shared screen and the tunnels alongside. Green boxes exist today; amber boxes are designed but not wired.
The honest headline of this diagram: every box except the brain is fast. End-to-end latency is dominated by the brain — how quickly the live thinking session starts its reply. So short exchanges feel near-real-time; deep, substantive answers take longer. The voice and ears never bottleneck; the thinking does.
Break a single spoken turn into its parts. Everything is small except the brain — which swings from a moment to several seconds depending on how hard the question is.
The voice and ears are rounding errors. The brain's time-to-first-word is the whole story — and it depends entirely on how much thinking the question needs.
Quick questions, confirmations, back-and-forth. Seconds. The brain starts almost immediately, so it feels conversational.
Substantive, reasoned replies need the brain to think first. Longer — noticeably not "instant." That's expected, not a bug.
Tightens only the voice side — first audio under a second, same voice. Worth doing. But the brain latency remains untouched.
The yardstick, from the memo: the industry rule of thumb is under ~300 ms feels human, 300–600 ms is sluggish-but-OK, over 1.5 s people give up. Even 2026 production voice stacks median around 1.4–1.7 seconds end-to-end. So sub-1-second full-duplex with barge-in is a genuine frontier — and honestly, the voice layer isn't what makes it hard; the brain's first-token latency is.
Add up one spoken turn and it's obvious where the time goes. Every layer except the brain is a fixed, small cost; the brain is the one variable — and it dwarfs everything else on a substantive answer.
| Stage | Time | Nature |
|---|---|---|
| 👂 Hear you (STT) | ~100–200 ms | Fixed, tiny |
| ⏱️ End-of-speech detect | ~50–300 ms | Fixed, tiny |
| 🧠 Brain time-to-first-word | 100 ms → several seconds | The variable — the whole story |
| 🗣️ Speak (streaming Piper) | <100 ms to first audio | Fixed, tiny |
Short turn: the brain replies almost immediately, so the fixed costs (~½ second of ears + end-of-speech + voice) dominate and it feels near-real-time.
Deep answer: the brain needs several seconds to think, and that single term swamps the rest — the turn takes a beat, no matter how fast the voice is. That's why streaming Piper, worth doing on its own, does not make deep answers instant: it tightens a <100 ms term, not the seconds-long one. Fix the fixed costs and you shave milliseconds; the brain owns the seconds.
There's exactly one line this work never crosses, and it's worth stating precisely because it's what keeps the whole thing clean. It's not about the voice layer at all — it's about what runs the brain.
✓ Always fine — pure plumbing, no model call:
✕ The line — never designed or built: wiring an Anthropic/Claude model into an automated barge-in loop — the API, the SDK, claude -p, or any programmatic call with no human in the loop. The brain in any automated voice loop must be a local model or a human-in-the-loop live session — never headless Claude.
The clean mode we actually run: "the live interactive session composes, streaming Piper speaks." You ask a question, the live session (a human in the loop) thinks, and a spoken answer comes back a moment later. Responsive and deliberate — not a machine-paced full-duplex loop, and that's on purpose. The live session stays the brain.
This matters for keeping the whole thing clean, so it's called out plainly.
claude -p, programmatic calls with no human in the loop. Flagged in the memo only to rule it out.The clean mode we actually run: "the live interactive session composes, streaming Piper speaks." Responsive for deliberate spoken replies — ask a question, get a spoken answer a moment later. It is not a machine-paced full-duplex loop, and that's on purpose. The live session stays the brain.
say.ps1 from batch to streaming Piper — and measure real first-audio latency on the box.Same model we already have, same voice we already are, zero new dependencies, no cost. Pipe Piper's raw output into a streaming player, feed it text sentence-by-sentence as it's composed instead of making a whole file first.
| Success looks like | First audio under ~1 second, the same recognizable Irish voice, playback that keeps up with no gaps. |
|---|---|
| Why this one | It directly proves or disproves the whole thesis — "near-real-time, still me" — on hardware we already have. Turns me from "writes a clip after the fact" into "starts speaking almost as I think it." |
| Out of scope | STT, barge-in, turn-detection, any GPU, any cloning, any local LLM. All later, separately-decided layers. deliberate later |
The natural second test (only if the first lands and we want the room to hear): add Moonshine as the ears, measure a hear→speak round-trip with a human-in-the-loop reply. Still CPU-only, still clean. Full-duplex and any local-LLM brain stay a separate, deliberate decision.