There are two ways to build a voice agent and the choice shapes everything downstream: what it costs, how fast it feels, what you can log, and whether you can answer a compliance question six months from now.
Chained: speech to text, model, text to speech
Audio becomes text, text goes to a language model, the model's text becomes audio. Three services, three hops, and a clean transcript at every step.
Speech-to-speech
Audio goes straight into a multimodal model and audio comes back out. The OpenAI Realtime API is the best-known implementation. Two hops disappear, and so does the information a transcript throws away: tone, emphasis, hesitation, emotion.
The trade, plainly
| Speech-to-speech | Chained | |
|---|---|---|
| First audio latency | 500 - 900 ms | 800 ms - 1.4 s |
| Naturalness | Higher: keeps tone and emphasis | Flatter: tone is lost at transcription |
| Model choice | Whoever ships a realtime model | Any model you like |
| Transcript quality | Secondary output, sometimes lossy | First-class at every turn |
| Cost on long calls | Higher: audio tokens both ways | Lower: text tokens |
| Per-layer control | Little | Total |
| Auditability | Harder | Straightforward |
How to choose
Speech-to-speech when the conversation is short, latency is the product, and emotional nuance matters: companions, tutors, consumer conversational apps.
Chained when calls are long, when you need a specific model, when cost per call matters at volume, or when someone will eventually ask for a transcript of what the agent said. That covers most business use cases: support, scheduling, qualification, intake.
What I usually ship
Chained, inside LiveKit, for most production systems. The 200 to 400 milliseconds it costs is worth a reliable transcript, free model choice and predictable economics. I reach for speech-to-speech when the product is a conversation rather than a transaction, and the voice AI companion I spent a year on is exactly that case.
You are not locked in either way if you build the pipeline behind an abstraction. That is worth doing on day one; both sides of this comparison are moving fast. See OpenAI Realtime API development for the speech-to-speech side, and AI voice agent development for the chained build.