The vendor's demo ran for forty minutes and did not put a foot wrong.
Every example on the screen had been chosen by the vendor's own people: a clean case, an obvious call, an easy win. The tool caught each one, and the room nodded along.
Nobody in that room had in front of them the pile of real cases waiting back at the plant — the borderline ones, the ones two experienced people would argue about over the same photograph or the same ticket. Nobody had agreed, before the demo even started, what a correct answer looked like for those.
A few months after the system went live, it started disagreeing with the person on the floor. Not often — maybe one case in twenty. But there was no way to say which of them was right, because the question had never been written down anywhere. The vendor's number from the demo was still the only number anyone had, and it had never once been tested against a case from this plant.
Why we build it this way
The mistake is rarely choosing the wrong tool. It is agreeing to compare tools before anyone has written down what a right answer looks like for this business, on the cases that are actually hard. A set built from easy examples makes almost anything look competent, because easy examples are easy for every approach — including the one you do not end up choosing.
The obvious alternative is to pick whatever looks strongest on paper, put it to work, and measure once it is already carrying real decisions. That gets the order backward. By the time the first disagreement shows up, the system has a workflow built around it and a team that has started trusting its output. Fixing a mismatch at that point means unwinding something live, not choosing between two options on a table.
So we insist on the plainer order: sit with the people who do the job, collect the cases they argue about with each other — not the average case, the disputed one — and agree on what counts as right for each of them, before any approach is chosen. That agreement becomes the one thing every option has to answer to.
A set built from easy cases tells you that something works. A set built from the cases your own people argue about tells you whether it is right.
It also changes what the comparison actually is. It is no longer a vendor's number against our number. It is every option measured against the same fixed yardstick, built from this business and owned by it — which is the only way a plant head can trust the answer without having to trust whoever is selling it.
How it works, roughly
Before we propose an approach, we build the test set with the people who will live with the decision. It holds:
- The cases two experienced people in the room would disagree about, not the ones everyone already agrees on
- A written answer for each one, signed off by whoever is accountable for that call today
- A run of whatever you use now, scored against that same set, so the comparison starts from where you actually stand
Only once that set exists do we propose anything, and only measured against it — not a public leaderboard, not the vendor's own sample. Every approach under consideration is scored on the same cases, so the choice is made on this business's terms, not on how a demo happened to be lit.
When something disagrees with the set after it is live, the disagreement is raised to whoever signed off on the set in the first place, not quietly absorbed as probably fine. The set can be changed, but only by that same written decision — never by whoever happened to be annoyed that week.
A year on, the test set is usually worth more than whatever we built against it. It outlives the vendor, the model, and eventually us, and it can judge whatever comes next on the same terms. A person still owns what counts as right, and the record of every case lives in your building, not ours.
See how we sequence a programme →