Blog Details

Once you've measured where latency lives in your voice agent pipeline (see our companion article on latency measurement), the next step is fixing it in priority order. Not all optimizations are equal — some cut hundreds of milliseconds, others save single-digit milliseconds for real engineering effort. This article ranks the techniques we've found to matter most, roughly in order of impact-to-effort ratio.

Tier 1: Stream Everything (Highest Impact, Moderate Effort)

If you're not already streaming at every stage, this is the first thing to fix — it typically has the largest single impact on perceived latency.

• Stream STT partial results into the LLM as the user speaks, rather than waiting for end-of-speech plus full transcription.
• Stream LLM tokens into TTS as they're generated, rather than waiting for the complete response before starting synthesis.
• Stream TTS audio out in chunks as it's synthesized, rather than waiting for the complete audio clip.

# Before: blocking, sequential
transcript = stt.transcribe_full(audio)          # wait for full STT
response = llm.generate_full(transcript)          # wait for full LLM response
audio_out = tts.synthesize_full(response)         # wait for full TTS
play(audio_out)

# After: streaming, pipelined
async for token in llm.generate_stream(transcript):
    async for audio_chunk in tts.synthesize_stream(token):
        play_chunk(audio_chunk)

The end-to-end latency win from this alone is often 500ms-1s+ for longer responses, because synthesis and playback of the first sentence overlap with generation of later sentences instead of happening strictly after.

Tier 2: Reduce Time-to-First-Token (High Impact, Model-Dependent Effort)

• Use a smaller/faster model for the first response chunk, then optionally hand off to a larger model for complex follow-up — a "fast first word" pattern, sometimes implemented as a small model generating an acknowledgment ("Let me check that...") while a larger model or tool call runs in parallel.
• Reduce prompt size. Long system prompts and conversation history directly increase TTFT. Summarize older turns instead of replaying full history every request.
• Choose inference infrastructure with lower queuing/cold-start latency. Dedicated or reserved-capacity endpoints avoid the P95 spikes common on shared serverless inference during traffic bursts.

Tier 3: Parallelize Tool Calls With Speech (High Impact, Moderate Effort)

If a user's request requires a tool call (database lookup, API call) before you can give a real answer, don't sit in silence during that call. Two patterns:

Acknowledgment-then-continue:

speak_immediately("Let me check that for you.")
result = await tool_call()
speak(format_response(result))

Speculative parallel execution: kick off likely tool calls as soon as intent is detected from partial STT, before the user finishes speaking, and use the result if the guess was correct (discard and fall back if not). This is higher effort and only worth it for tool calls with meaningful latency (300ms+) and high-confidence intent detection.

Tier 4: Trim STT and VAD Latency (Moderate Impact, Low-to-Moderate Effort)

• Tune voice activity detection (VAD) end-of-speech timeout. A shorter timeout reduces perceived latency but risks cutting users off mid-sentence; a common production value is 500-700ms of silence, tuned against real usage data rather than guessed.
• Use a streaming STT provider/model instead of batch transcription, so partial transcripts are available well before the user stops speaking.
• Co-locate STT infrastructure with your application server region — a cross-region round trip adds fixed latency to every single turn.

Tier 5: TTS-Specific Optimizations (Moderate Impact, Low Effort)

• Synthesize in small chunks (sentence or clause-level) rather than waiting for a full paragraph — pairs with Tier 1 streaming.
• Cache common responses (greetings, common confirmations, error messages) as pre-synthesized audio instead of calling TTS live for text that never changes.
• Pick a TTS model/provider with lower per-request overhead if your current one has high fixed latency per call, independent of streaming support — benchmark this directly rather than assuming.

Tier 6: Infrastructure and Networking (Lower Impact per Change, Cumulative Effect)

• Keep persistent connections (WebSocket/gRPC streams) to STT/LLM/TTS providers instead of establishing new connections per turn — TLS handshake overhead adds up across many short turns.
• Co-locate all pipeline components (STT, LLM, TTS, application server) in the same region/availability zone to minimize network round trips between stages.
• Right-size timeouts and retries — an overly aggressive retry policy on a slow-but-working call can make an already-slow interaction worse; an overly patient timeout on a genuinely failed call wastes the user's time waiting for a failure that should have surfaced faster.

A Prioritized Rollout Plan

If you're starting from a non-streaming, non-optimized baseline, this is the order that tends to produce the best return on engineering time:

1. Implement end-to-end streaming (STT partials → LLM tokens → TTS chunks). Biggest single win.
2. Add acknowledgment-then-continue for any tool call over ~300ms.
3. Trim system prompt size and conversation history sent per turn.
4. Tune VAD end-of-speech timeout against real usage data.
5. Cache static TTS responses.
6. Evaluate faster STT/LLM/TTS provider options with a proper benchmark harness (see our latency measurement article) rather than switching on vendor claims alone.
7. Address infrastructure co-location and persistent connections last — real but smaller, cumulative wins.

Measuring the Payoff

After each change, re-run your fixed benchmark set and compare P50/P95 per stage, not just the aggregate. A change that improves P50 but worsens P95 (for example, a smaller model that occasionally needs a retry) can make the product feel less reliable even as the average looks better — track both, and weight P95 more heavily for anything users perceive as "occasionally broken."