Almost every voice agent that fails in production fails on latency, and almost every team debugging it starts in the wrong place. They tune the prompt or swap the model, when the time is usually being lost somewhere else entirely.
The thresholds that matter
Under 800ms: the conversation feels alive. This is the target.800ms to 1.5s: noticeably slow but usable. Most managed-platform agents live here.1.5s to 2.5s: callers start talking over the agent, and if barge-in is not handled, the call degrades fast.Over 2.5s: people assume the line dropped and hang up.The budget, stage by stage
| Stage | Typical | Achievable |
|---|
| End-of-turn detection | 300 - 700 ms | 150 - 300 ms |
| Speech to text (final) | 100 - 300 ms | 50 - 150 ms |
| Model time to first token | 300 - 900 ms | 200 - 400 ms |
| Text to speech, first audio | 150 - 500 ms | 80 - 200 ms |
| Network and transport | 50 - 200 ms | 30 - 80 ms |
| Total, first audio out | 900 ms - 2.6 s | 510 ms - 1.1 s |
Look at the first row. Turn detection is usually the largest single cost in the budget and the one nobody measures, because it does not appear in any vendor's latency benchmark. Vendors measure from end of speech; the caller experiences it from when they stopped talking.
Eight fixes, roughly in order of payoff
Tune turn detection against your own recordings. Default silence thresholds are set conservatively. A semantic turn-detection model beats a fixed timeout, especially for callers who pause mid-thought.Start the model call on interim transcripts, not final ones. Speculative generation costs you some wasted tokens and buys 200 to 400 milliseconds.Stream text-to-speech sentence by sentence. Do not wait for the full response before synthesising the first sentence.Stop resending the full conversation history every turn. On a long call this quietly becomes your dominant model cost and your dominant model latency.Choose providers on time-to-first-byte, not on quality demos. Voice providers differ by hundreds of milliseconds on first audio.Move retrieval off the critical path. Cache aggressively, start the lookup in parallel with the model's opening words, and have a fallback for when it is too slow to wait for. This is the hard part of RAG inside a voice agent.Co-locate your services. A model in one region and synthesis in another can cost 100 milliseconds for nothing.Use a filler phrase honestly. A short acknowledgement buys real time, but it becomes obvious and irritating if it fires on every single turn.Measure before you optimise
Instrument every stage separately and log per-turn timings on real calls, not on synthetic tests. Almost every latency problem I have been brought in to fix turned out to be in a different stage than the team assumed. Twice it was turn detection. Once it was a voice provider in the wrong region. Once it was 40,000 tokens of conversation history being resent on every turn of a twenty-minute call. If your agent is slow and you want it measured properly rather than guessed at, send me the details.
