On an appointment call, a quick "I'll check availability" can be followed by a long wait, a correction from the caller, and another lookup. The team reviewing that call needs to know where the conversation slowed down before deciding what to change.
What Is Voice Agent Latency?
Voice agent latency is the delay between a caller finishing a speaking turn and hearing the agent begin its response.
When comparing AI voice latency reports, check their start and end points. Some dashboards measure only part of the caller-heard interval defined here.
These terms help explain the measurements:
- Turn: one person's contribution to the conversation.
- Endpointing or end-of-turn detection: the system deciding that the caller has finished speaking.
- Barge-in: the caller interrupting the agent while it's speaking.
- PSTN: the public telephone network used for ordinary phone calls.
- Tool call: a request to another system, such as checking a calendar or updating a customer record.
Dial-to-answer time covers call setup and ringing. Answer-to-first-greeting time starts when the call is answered and ends when the caller hears the greeting begin. Neither is the same as the response gap after a question.
The recording location matters, too. Twilio's core latency guide distinguishes caller-perceived timing from measurements inside the platform. A platform metric may exclude part of the delivery path to the caller's phone.
Speed is one part of a natural exchange. An agent also needs to let callers finish, handle corrections, and give useful answers. Our guide to how AI voice agents work covers the broader call process.
What Causes Delays in an AI Phone Conversation?
An AI phone response can involve these stages:
- The caller's audio reaches the system.
- The system recognizes the speech and decides whether the caller has finished.
- The agent prepares a reply, sometimes after retrieving information or contacting a business system.
- The response becomes speech.
- The audio travels back and becomes audible to the caller.
Speech recognition is also called speech-to-text (STT) or automatic speech recognition (ASR). A large language model (LLM) can prepare the reply, and text-to-speech (TTS) converts it into audio. Turn detection may use the recognition system or a separate component.
Recognition can run while the caller is still talking. Also, a finalized transcript segment doesn't necessarily mean the complete turn is over. Deepgram's endpointing documentation explains that distinction for its streaming recognition system.
Two component measurements need particular care:
- Time to first token: the wait for the model's first piece of text, not spoken audio.
- Time to first audio byte: the wait for initial audio at a specified interface, not necessarily sound at the phone.
LiveKit's measurement reference defines these intervals. Waiting for a calendar or customer-record system adds another interval. After audio is generated, routing, format conversion, buffering, and playback can still add delay.
In an illustrative streaming workflow, stages may overlap:
- Recognition runs during the caller's speech.
- Turn detection considers incoming speech while recognition continues.
- Response planning starts when usable input is available.
- Speech generation starts before the entire reply is written.
- Delivery and playback start before all the audio is generated.
The critical path is the sequence of work that must finish before the caller can hear the response. Adding the full duration of every stage can count overlapping work twice. Don't add transcription wait separately if it's already included in an end-of-turn metric. LiveKit's latency guide describes one example of overlap: preemptive response generation can begin while the caller is still speaking.
Integrated speech-to-speech models may combine stages and expose different measurements. The caller-heard start and end points remain useful; the architecture alone doesn't establish the response time.
What Is a Good Latency for an AI Voice Agent?
A useful voice AI latency target depends on the measured interval, call type, phone route, and whether an external result is needed. A single cutoff doesn't account for all of those conditions.
Set a target you can test:
- Separate opening greetings, simple replies, and tool-dependent answers.
- Review representative calls with the people who will use or supervise the workflow.
- Agree on a response-time target and quality conditions for each call type.
- Test typical responses, slower responses, and failures against those conditions.
For example, waiting while a caller spells a surname may be appropriate. More aggressive turn detection risks cutting them off; more conservative detection can delay replies after clearly finished turns. Check both behaviors before changing the setting.
Keep a measured result, an engineering target, and a service commitment distinct. A measured result describes a particular test. A target describes what the team aims to achieve. A service commitment describes what the provider agrees to deliver.
The Twilio and LiveKit guides linked above explain measurement scope and engineering tradeoffs. Use that context to design tests, not to assume another configuration's result applies to yours.
Measure What the Caller Actually Hears
Write down what starts and stops each timer before comparing results. Ask the implementation team to identify where the events are recorded and how the timestamps line up.
Use this table as a measurement checklist:
| Measurement | Start | End | Evidence needed |
|---|---|---|---|
| First greeting | The call is answered, using a stated answer event | The caller hears the greeting begin | Answer event and caller-side audio on an aligned timeline |
| Ordinary reply gap | The caller finishes the complete turn | The caller hears the response begin | Caller-side audio or an equivalent capture with a known timing offset |
| Time to first useful answer | The caller finishes the request | The first task-relevant information becomes audible | Audio review with a written definition of useful |
| Tool-dependent reply | The caller finishes providing the required information or authorization | The result message begins audibly, after the system confirms it | Linked audio and business-system result records |
| Tool execution | The application sends the operation and required inputs | The connected system returns a result or error | Linked application and business-system records |
| Full task duration | The caller starts the original request | The confirmed outcome message finishes playing | The complete conversation and verified business result |
| Interruption stop delay | The caller begins a genuine interruption | The agent's old speech stops at the caller's phone | Caller-side audio, not just a detection event |
| Component timing | The documented component request or input event | The documented first token, audio chunk, or completion event | Logs that preserve the provider's measurement definition |
"Time to first useful answer" is a testing label used here, not a standard field every platform provides. Define it before reviewing calls. An available appointment time can be useful information, but it isn't a completed booking. Keep the completion timer separate.
Consider this illustrative sequence:
- The caller asks to move an appointment.
- The agent says it's checking availability.
- The calendar returns available options.
- The caller chooses an option.
- The calendar confirms the change.
- The agent tells the caller the confirmed appointment details.
Step 2 measures acknowledgement. Step 5 establishes the business result. Step 6 delivers that result to the caller. The difference between requesting an update and confirming it also matters for verified CRM updates.
For the tool-dependent reply, start timing after the caller's final choice or authorization. The full task duration starts earlier and includes the caller's speech and thinking time. It isn't a measure of the agent's response delay alone. If the system returns an error or no result, record that outcome rather than treating it as a successful completion.
Some timers above stop when a message begins; full task duration stops when the outcome message finishes. Keep those labels consistent.
Use one continuous caller-side timeline where practical. A provider-side recording may miss downstream carrier or handset delay. If records come from different systems, engineering needs to align their clocks and document any uncertainty. A hand-operated stopwatch can flag an uncomfortable wait, but not establish precise millisecond differences.
Use test accounts and non-sensitive data, and get organizational approval for the recording setup.
Read P50 and P95 Without Hiding Slow or Failed Turns
What P50 and P95 Tell You
P50 is the median, the middle of a defined set of measurements. P95 is the value at or below which roughly 95% of those measurements fall. Tied values and calculation methods can affect the exact split. AWS explains percentiles in its CloudWatch statistics definitions.
P95 latency isn't the worst response, an accuracy score, or a guarantee about future calls. A good median can coexist with slow responses elsewhere in the sample. A small test can reveal problems without establishing a stable picture of production performance.
State whether you're measuring turns or calls. When all measured turns are pooled, calls with more measured turns contribute more observations. Report both the number of calls and the number of measured turns.
Don't average separate P95 values to calculate an overall P95. Prometheus's histogram guidance explains why averaging precomputed quantiles is invalid. Use comparable underlying observations or an appropriate combined distribution.
Don't add component P95s and label the sum end-to-end voice latency, either. This follows from the mathematics of percentiles: the slowest component observations can occur on different turns. Measure each complete interval, then calculate its percentile.
A Reusable Reporting Card
Use this template to record your test.
| Field | Record |
|---|---|
| Test scope | Workflow, test dates and timezone, configuration/version |
| Call conditions | Phone or browser route, known region/carrier, language, device, noise, load, first-start or warmed-up state |
| Metric | Exact start, end, recording location, units, clock alignment |
| Sample | Calls attempted, eligible turns, successfully timed turns |
| Results | P50, P95, calculation method and any approximation |
| Exceptions | No-audio responses, timeouts, incomplete calls, interruptions, exclusions and reasons |
| Quality | Premature replies, missed corrections, verified task completion, handoff outcome |
| Acceptance | Agreed conditions, pass/fail or insufficient evidence, owner and next test |
Separate ordinary replies from tool-dependent turns, and browser tests from phone calls. Identify first-start calls separately from calls made after the system is already running.
If percentiles cover only successful responses, label them that way and show failures beside them. Never record a timeout as zero latency or drop it silently. Use "unavailable" when a timing couldn't be measured.
Record premature replies and overlapping speech separately. If the agent starts before the caller finishes, turning that negative speech gap into zero hides the interruption instead of measuring a successful response.
Test What Happens When the Caller Interrupts
Barge-in needs three checks:
- Did the system recognize the interruption?
- Did the old response stop playing to the caller?
- Did the agent use the correction and continue the task correctly?
Here's a hypothetical appointment exchange:
Agent: "I can offer Thursday at..."
Caller: "Actually, Friday afternoon."
Agent: "I'll check Friday afternoon instead."
The agent should stop reading Thursday's option and use Friday in the next lookup. It shouldn't announce a changed booking until the calendar confirms the change. Our guide to reliable appointment changes covers the wider workflow.
Retain evidence of the caller's speech start, interruption detection if available, any request to stop playback, the last audible old speech, the recognized correction, and the final calendar state. LiveKit's turn-handling documentation explains interruption handling and brief listener acknowledgements, sometimes called backchannels.
Test a pause during spelling, background noise, and a brief "mm-hmm" separately. None automatically means the caller wants the agent to stop.
Stopping speech also doesn't automatically cancel a pending business action. LiveKit's tool documentation describes this distinction. Check the action's status before proceeding. Retrying a write without knowing whether it succeeded can create a duplicate.
For implementers: a playback acknowledgement isn't proof that the caller heard the words. Twilio's Media Streams reference explains that mark messages can acknowledge cleared audio as well as completed playback. Interpret them alongside any requests to clear the audio buffer.
Run a Real-Phone Acceptance Test
An operations lead can organize this test with an implementation partner without writing code:
- Choose a representative workflow. Define success, timing targets, and fallback behavior before testing.
- Use the intended phone number and calling path. Record known routing details and mark unknown ones as unknown.
- Run consistent normal and difficult scripts under representative caller conditions.
- Match caller-side audio with transcripts and tool results where available.
- Repeat under expected busy conditions. Compare equivalent groups of calls before and after a change.
- Review timing and task outcomes together. Retest failures before expanding traffic.
Use the following matrix to plan the calls:
| Test | Expected behavior | Evidence to retain |
|---|---|---|
| First call after idle | Greeting begins within the agreed opening target | Answer and greeting timestamps, startup condition |
| Simple knowledge question | Correct, concise answer without avoidable silence | Caller-heard gap and answer review |
| Name or number with a pause | Caller can finish; complete value is captured | Audio and captured value |
| Caller corrects the agent | Old speech stops and the correction changes the task | Audible-stop timing, corrected fields, final state |
| Slow or failed lookup/update | Accurate progress message, no false success, defined recovery | Acknowledgement, system result/error, final outcome |
| Noise, speakerphone, or supported language variation | Useful input is retained without excessive false interruptions | Environment, transcript, interruptions, requests to repeat |
| Expected peak traffic and relevant phone routes | Timing and task outcomes meet agreed conditions | Route-specific samples, load, P50/P95, failures |
| Human fallback | Caller reaches the defined next step with useful context | Transfer/fallback outcome and information received |
Distinguish a successful transfer from a person actually answering. Hold time belongs in the caller's experience, but not in a model-processing metric.
To accept a workflow, check that timings meet the team's declared targets and that speed hasn't come at the expense of complete input, correct actions, or reliable recovery. Investigate an unconfirmed booking or lost correction even if the overall P95 passes. A small pilot helps diagnose problems; it doesn't guarantee reliability.
A browser demo is useful for checking a flow, but it doesn't validate every phone route. Twilio's media-edge documentation shows how its observation points differ between browser/SDK and carrier calls. That supports testing the actual route, not assuming every PSTN call adds the same delay or uses the same audio format.
This is a suggested test procedure, not an industry certification or a report of tests we've run.
How to Reduce Voice Agent Latency Based on the Evidence
Use the symptom to choose an investigation, not to assume a cause:
| Symptom | Check next | Possible adjustment and tradeoff |
|---|---|---|
| Only the greeting is slow | Startup and answer-to-greeting records | Review initialization or prepare resources before calls arrive; keep this separate from later reply delays |
| Agent waits after clearly finished turns | Turn-end decision versus actual speech end | Tune detection using pauses and spelling tests; faster settings can cause cutoffs |
| Response generation takes most of the time | Prompt, model, provider, and first-token timing | Remove unnecessary context or test another model while preserving instructions and accuracy |
| Audio exists at the server but arrives late | Output buffering, routing, conversions, and recording location | Review the delivery path without sacrificing intelligibility |
| Only tool-dependent turns are slow | Operation sent, system result returned, spoken result delivered | Remove redundant calls or run safe independent work together; preserve ordering and verification for updates |
| Slow responses increase at busy times | Comparable calls, resource queues, routes, and external-system load | Test capacity or location changes; don't diagnose from the average alone |
| Interruptions are detected but old speech continues | Buffered audio, stop request, and caller playback | Fix stopping and recovery, then verify the next turn |
For example, a faster model won't address a delay that occurs after audio has already been generated. If simple answers are timely but appointment lookups are slow, inspect the calendar request and result delivery before changing every response setting.
Make one clearly identified change, repeat comparable tests, and retain it only when timing improves without an unacceptable loss in task quality. The LiveKit and Twilio latency guides explain the components; your call records show which needs attention.
Test Your Workflow With Airsound
At Airsound, our preview calls let teams test intent detection, knowledge responses, and handoff rules before launch. Our visual builder connects call routing and detail collection with actions such as SMS, CRM updates, and human handoff. We support low-latency concurrent calls, natural turn-taking, and barge-in handling.
Use preview calls to check the workflow, then test the phone route your callers will use. Transcripts, outcomes, and workflow events are available for review and export. Fallback paths can include a transfer, ticket, summary, or follow-up. The reporting card above is a proposed test template. Work with our team to confirm the available timing records, recording setup, and reporting requirements for your implementation.
Request a demo with three scripts: a normal call, an interruption, and a slow-tool scenario. We'll discuss how to test that workflow and what evidence to collect before expanding it.
