Bake-off (vendor evaluation)

What it is

A bake-off puts two or more vendors through an identical test. The test material is yours: golden recordings of real calls for speech accuracy, a demo script with scripted traps for conversation quality, a sandbox of your system for integration, and, at larger scale, a synthetic caller that runs the same personas against every agent. Scoring is written down before the first call. Every vendor gets the same information and the same amount of tuning time.

The word comes from cooking contests; in procurement it has come to mean any structured comparison, but the value is in the structure, not the label.

Why it matters when buying

Vendor demos are optimised to look good; a bake-off is optimised to tell vendors apart. It surfaces the differences that matter for your callers (accent coverage, tool-turn latency, behaviour under interruption, what the agent does when it does not know) and produces evidence you can show a procurement committee or a regulator. It also reveals how each vendor works with you under mild pressure, which predicts the implementation.

What to ask

Before you start, fix the script, the recordings, the scoring sheet and the pass/fail criteria for each trap, and share them with every vendor. Insist that calls are placed by your people from real phones. Report medians and 90th percentiles, not a single best run. Give every vendor a written right of reply to their scores. The demo guide provides scripts and scoring sheets you can use as they are.