The voice AI latency checklist

1. Voice activity detection thresholdToo conservative and you wait after the user finishes; too aggressive and you cut them off mid-sentence.
2. Endpointing silence durationThe gap required before a turn is considered over — tune per language, not once for English and reused everywhere.
3. Network jitter and packet loss handlingA pipeline tested only on office wifi hides this entirely until a real mobile network exposes it.
4. Transport choiceWebSocket versus WebRTC changes the latency floor before any application code runs.
5. Speech-to-text streaming vs. batchWaiting for a full utterance before transcription starts adds the entire utterance length to every turn.
6. Time to first token from the LLMThe gap between sending a prompt and receiving the first streamed token — not the same as total generation time.
7. Prompt and context lengthA longer context window measurably slows first-token latency on most providers.
8. Tool calls and retrieval stepsEach round-trip to a tool, database or retrieval index adds serial latency unless deliberately parallelised.
9. Text-to-speech time-to-first-audioSome providers synthesise the whole response before playback starts; streaming synthesis avoids this.
10. Audio buffering on the clientOverly large client-side buffers add latency invisibly, trading a small amount of choppiness for a lot of delay.
11. Cold startsA serverless or scale-to-zero deployment adds seconds to the very first turn after idle — measure p99, not just p50.
12. Geographic distance to the inference regionA model hosted in one region adds real round-trip time for a user on another continent — this is why "it works fine for me" is not a benchmark.

Measure each stage separately before optimising any of them — the biggest gain is rarely where it's assumed to be.