A passing transcript is not a working AI call
The short answer
If your QA process only reads the transcript, you are scoring a chatbot. An AI call agent is a live pipeline: speech-to-text, a model that decides what to say and do, and text-to-speech, all running while the caller can talk over it. Each stage fails in a way a text test never sees. [SRC001]
A passing transcript is not a working call. [SRC001]
Why transcript QA misses the failures that cost money
Vervali's 2026 checklist is blunt about the stack. A misheard account number, a half-second of dead air, an interruption the agent speaks over, or a backend action it never triggers will not show up as a clean script pass. The dimensions that decide production readiness—recognition accuracy, latency, barge-in, task completion, faithfulness, tool-calling, and safety—sit almost entirely outside traditional QA scripts and text-only LLM evals. [SRC001]
Hallucination is not a side issue on voice. NIST's generative-AI profile calls it confabulation: “the production of confidently stated but erroneous or false content.” A voice agent speaks those assertions aloud. [SRC005] [SRC001]
Score accuracy and experience, not just task completion
ServiceNow's EVA framework, published on Hugging Face in March 2026, argues that voice agents have to satisfy two objectives at once: complete the task correctly and faithfully, and do it in a way that works as spoken conversation. EVA reports two composite scores: EVA-A for accuracy and EVA-X for experience. [SRC002]
EVA-A covers task completion against a verifiable end state, faithfulness to policy and tool outputs, and speech fidelity on spoken entities such as confirmation codes and amounts. EVA-X covers conciseness, conversation progression, and turn-taking. Across 20 cascade and audio-native systems, the authors found a consistent accuracy-experience tradeoff: agents that do well on task completion tend to deliver worse user experiences, and vice versa. No single configuration dominated both axes. [SRC002]
Named-entity transcription was a dominant failure mode. A single misheard character can cascade into an authentication failure and a full conversation breakdown. [SRC002]
On the BPO floor, sampling 2% of calls is the other failure
Pre-launch simulation is necessary. Production still needs 100% coverage. Qualia's homepage describes the audit bottleneck as “only 5% of calls audited – reviewed manually by ear,” and the CallPulse FAQ says manual teams typically review about 2% of calls. [SRC004] [SRC003]
CallPulse is Qualia's answer on the product page: analyze 100% of calls, score openings, compliance, objection handling, empathy, and fatal-error checks, and support Hindi, English, and Hinglish operations. [SRC003] Pair that with Voice Assistant when the same floor is also running AI callers, so QA and the agent share one conversation record.
A practical 2026 rubric
- Recognition: did STT hear names, numbers, and corrections under noise and Hinglish?
- Latency and barge-in: did the agent answer in time and stop when interrupted? [SRC001]
- Task completion: did the CRM or policy end-state actually change? [SRC002]
- Faithfulness: did spoken facts match policy and tool output, with no confabulation? [SRC005] [SRC002]
- Speech fidelity: were confirmation codes and amounts spoken correctly? [SRC002]
- Coverage: was every call scored, not a 2% sample? [SRC003]
FAQ
Why isn't a transcript enough to QA an AI voice agent?
A voice agent is a live speech-to-text, dialog, and text-to-speech pipeline. Errors such as misheard numbers, dead air, talking over the caller, or a wrong spoken confirmation code never show up as a clean transcript pass. Vervali's 2026 checklist puts it directly: a passing transcript is not the same as a working call.
What should BPO QA measure besides script adherence?
Score both accuracy and experience. ServiceNow's EVA framework splits this into EVA-A (task completion, policy faithfulness, spoken-entity fidelity) and EVA-X (conciseness, conversation progression, turn-taking). On live floors, also score 100% of calls against your rubric instead of a tiny manual sample.
How does CallPulse fit this?
Qualia's CallPulse product page says it analyzes 100% of calls, scores against a QA rubric, and is built for Hindi, English, and Hinglish operations—replacing the 2% a manual team typically reviews.
Sources
- Testing AI Call Agents: What QA Actually Has to Cover in 2026 — Vervali
- A New Framework for Evaluating Voice Agents (EVA) — Hugging Face / ServiceNow AI
- CallPulse — AI Call Auditing for BPOs — qualiabits.com
- Qualia Bits homepage — qualiabits.com
- NIST AI 600-1: Generative AI Profile — NIST
Evidence map
- An AI call agent is a real-time STT, dialog-model, and TTS pipeline, and each stage fails in ways a text test never sees.
Evidence: SRC001 - NIST defines confabulation as the production of confidently stated but erroneous or false content.
Evidence: SRC005, SRC001 - ServiceNow's EVA framework jointly scores accuracy (EVA-A) and conversational experience (EVA-X) and reports a consistent tradeoff between the two.
Evidence: SRC002 - Qualia's CallPulse product page states that it analyzes 100% of calls versus about 2% for manual teams, with Hindi, English, and Hinglish support.
Evidence: SRC003, SRC004