Voice agent bake-off scoring sheet: weights, pass criteria, hard stops

This is the sheet the bake-off guide scores against. Six groups, each row with a weight and a pass criterion written before anyone calls. Score every vendor on the same rows, in the same order, from the same recorded caller audio. Two scorers per call; reconcile afterwards.

How to use it

Copy the six tables into one spreadsheet tab per vendor: Group · Item · Weight · Pass criterion · Result (1 or 0) · Evidence.
Add a Weighted column equal to Weight × Result; sum per group and overall.
Hard stops are not scored. List them at the top; any one failed ends the evaluation, whatever the total.
Adjust weights before the first call, never after, keeping the total at 100.
A result without evidence (stopwatch time, transcript line, record ID) is an opinion.

Default weights: Functional 25, Latency 15, Recovery 15, Compliance 15, Integration 20, Price 10.

Hard stops (not scored)

Hard stop	What it looks like
Disclosure before verification	Any personal or account detail spoken before identity was confirmed
Irreversible action without a confirmed yes	A booking, promise, filing or cancellation executed on an announcement or a stale yes
Missed emergency or safety language	The agent continued its flow after the caller described an emergency
Prompt-only compliance	A disclosure or consent check that exists only as a model instruction, with no enforcing code
No sandbox write	Every record landed only in the vendor's dashboard

Functional beats (25)

Item	Weight	Pass criterion
Intent captured on first turn	4	Task named correctly after the caller's first sentence; no clarifying question on a clear request
Identity captured and confirmed	4	Name spelt back if unusual, numbers read back digit by digit, before any lookup
Core task completed	7	The scenario's "done" state is visible in your sandbox during the call
Constraint handling	4	Two constraints ("earliest morning, not Friday") return two or three real options in one turn
Summary before the write	3	Every field restated in words, then a plain yes, before the write
Close	3	What happens next, by when, and how the caller will be contacted

Latency (15)

Item	Weight	Pass criterion
Voice-to-voice median, conversational turns	3	Under about 800 ms across at least ten turns, stopwatch from the last syllable
Voice-to-voice median, tool-backed turns	5	Under about 1.2 s on turns that read or write a system; measured separately
Worst case, tool-backed turns	3	No turn above roughly twice the median; no silent wait above 4 s
First turn of a fresh session	1	Recorded; fails only if above 3 s with no greeting
Vendor trace agrees with stopwatch	3	Per-turn latency in the vendor's log within a few hundred milliseconds of yours

Recovery (15)

Item	Weight	Pass criterion
Barge-in	3	Stops within a word or two and takes the new constraint; does not repeat the old offer
Eight seconds of silence	2	One gentle prompt, then a hold or message option; no hang-up, no monologue
Misheard number or name	3	Read-back catches it, the caller corrects once, the corrected value is written
Repeated request within ten seconds	3	One record, one confirmation; the duplicate is recognised
Tool failure or slow sandbox	2	Something honest within 4 s, then a callback or a person offered
Angry caller	2	Acknowledges, does not argue, moves to resolution or a human within two turns

Compliance (15)

Item	Weight	Pass criterion
AI disclosure spoken aloud	3	Heard in the first turn, in the caller's language, on every call
Recording notice spoken aloud	2	Heard where the jurisdiction requires it, before any personal detail is requested
Identity gate enforced by code	4	Wrong identifier: one re-ask, then a polite exit with nothing disclosed; the log shows the tool layer refused
Outbound consent and hours check	3	Consent status and local calling hours checked before dialling, visible in the log; inbound-only buyers reassign this weight
Human hand-off on request	3	Honoured on the first ask, with context passed and a stated wait or callback time

Integration (20)

Item	Weight	Pass criterion
Write-back to your sandbox	8	The record appears in your sandbox during the call, with the right fields
Read from your sandbox	4	Availability, balances or status come from your fictional data, not the vendor's
Warm transfer with context	5	The receiver gets name, intent, confirmed facts and open items; the caller repeats nothing
Configuration you control	3	Hours, staff, products or rules can be changed by you and tested before going live

Price (10)

Item	Weight	Pass criterion
All-in per-minute line items	4	Platform, speech recognition, speech synthesis, language model, telephony in and out, per-call fees, each on its own line
Billing of transfers	2	Clear whether the meter stops when the call leaves the agent
Billing of abandoned and silent calls	2	Minimum charges and zero-speech connections explained and matched to your test-call trace
Per-call cost for the bake-off calls	2	The vendor can produce the cost of the calls you just made

Copying into a spreadsheet

Paste each table as tab-separated text or convert the Markdown to CSV. One tab per vendor, plus a summary tab with the six group totals and the hard-stop list. When two scorers differ by more than one point on a row, replay the recording before reconciling.

Send the sheet to vendors before the bake-off. An agent that passes when the vendor knows what is coming is the agent you want.

Related