What to measure when you evaluate a voice agent

What to measure when you evaluate a voice agent
AI voicecall QAvoice agentsAEO

The short answer

A voice agent is not a chatbot with speakers. It is a live STT → model → TTS pipeline under a latency budget, while the caller can talk over it. If your eval only reads the transcript, you are grading a different product. [SRC001] [SRC002]

Score four layers: infrastructure (latency, ASR/TTS), execution (task success, tools, faithfulness), user behavior (barge-in, turn-taking), and business outcomes (containment, FCR). Then score 100% of production calls, not a sample. [SRC001] [SRC005]

Latency: percentiles, not averages

Callers feel Time to First Audio (TTFA)—from end-of-speech to the agent's first audible word. Hamming's guide treats under 800ms as the natural band and flags P50 above ~1.5s as a warning on cascading stacks. Track P50/P95/P99; a fine average can hide a brutal tail. [SRC001]

Break the stack: STT, LLM time-to-first-token, TTS time-to-first-byte, network, and turn detection. Fix the slow stage, not the dashboard average. [SRC001] [SRC002]

Barge-in and turn-taking

Interruptions are normal on phone. Measure true positives when the caller really interrupts, false positives from noise, false negatives when the agent talks over them, and how fast TTS stops (target under 200ms from speech onset). [SRC001]

ServiceNow's EVA work splits the same idea into two scores: EVA-A for accuracy (task completion, faithfulness, speech fidelity) and EVA-X for experience (conciseness, progression, turn-taking). A call can be “correct” and still feel broken. [SRC003]

Speech accuracy without worshipping WER

WER remains useful for ASR health, especially under noise and accents. It is not enough for money fields: names, amounts, OTPs, and confirmation codes need entity-level checks. Pair WER with faithfulness and tool-call success so a confident wrong number cannot pass as a good turn. [SRC001] [SRC002] [SRC003]

Task success and production census

Pre-launch simulations catch regressions. Production still needs a census. Use the same rubric on AI and human calls so model drift shows up as QA, not as a surprise CSAT dip. That is the operating model behind CallPulse 100% scoring and Qualia Voice on the same conversation record. [SRC005] [SRC004]

If you are still only reading transcripts, start with our note on why a passing transcript is not a working AI call.

Checklist to ship this week

  1. Instrument TTFA + end-to-end P50/P95 per component.
  2. Add barge-in true/false positive/negative rates and suppression latency.
  3. Score task success, faithfulness, and spoken entity fidelity—not transcript pass alone.
  4. Regression-block prompt changes that move those metrics outside tolerance.
  5. Score 100% of production calls; reserve humans for fails and outliers. [SRC001] [SRC005]

FAQ

What latency should a phone voice agent hit?

Hamming's production guidance (from 10M+ minutes protected) targets P50 under 1.5 seconds and P95 under 3.5–5 seconds end-to-end, with Time to First Audio ideally under 800ms. Averages hide bad tails—track percentiles.

Is Word Error Rate enough?

No. WER is useful for ASR health, but it treats every word equally and misses barge-in, latency, faithfulness, and entity errors on names and amounts. Pair WER with entity accuracy and conversation metrics.

How do I score barge-in?

Track true-positive interruption detection (target >95%), false positives from noise (<5%), false negatives (<5%), and time from speech onset to TTS suppression (target under 200ms).

Sources

  1. How to Evaluate Voice Agents: Complete Framework for Testing & Monitoring — Hamming AI
  2. Voice AI Evaluation Framework: Metrics & Tooling — Prodinit
  3. EVA: Evaluating Voice Agents — ServiceNow / Hugging Face
  4. Qualia Voice Assistant — qualiabits.com
  5. CallPulse — AI Call Auditing for BPOs — qualiabits.com

Evidence map

  • Production voice agents should track TTFA under ~800ms and end-to-end P50 under ~1.5s, with P95 watched separately.
    Evidence: SRC001
  • Barge-in quality needs detection accuracy (~95%+), low false positives/negatives, and sub-200ms TTS suppression.
    Evidence: SRC001
  • Voice evaluation should separate accuracy (task completion, faithfulness, speech fidelity) from experience (turn-taking, conciseness).
    Evidence: SRC003
  • Measuring only LLM text output misses most of what callers experience on a live voice stack.
    Evidence: SRC002, SRC001
See Qualia Voice