Two years after the first system went in, a different vendor came to pitch its replacement.
The deck was clean and the number on the last slide was good — their number, from a demo built on cases their own team had picked. The plant's quality lead let it run to the end. She had sat through decks like this before, and she was not there to argue with someone else's sample.
When the vendor finished, she opened a spreadsheet of her own. Forty-odd cases, each one a part her people had genuinely disagreed about on the floor years earlier, each with a written answer the team had agreed on and signed off at the time. She asked the vendor's engineer to run his tool against those forty cases, before anyone discussed price.
He had not planned on being marked against a test he had not written. A handful came back wrong — not a disaster, but not the clean sweep the deck had promised either. The meeting carried on, but on different terms than it had opened with.
Why we build it this way
Most of what a technology engagement produces loses value the day the next vendor shows up. The dashboard ages, the model gets superseded, the team that built it moves on to another account. One thing does not age the same way: a written record of what the plant's own people agree a right answer looks like, for the cases that are genuinely hard to call.
The obvious alternative is to let whoever is selling the next system set the terms of the comparison — their demo, their sample, their definition of good. That is usually not dishonesty, just that nobody grades their own homework fairly, us included. A vendor's number tells you how their tool performs on the cases the vendor chose. It says nothing about the case two of the plant's own best people would still argue about.
So when we build a test set with a customer, we treat it as separate from whatever we go on to build against it. It belongs to the plant, not to the engagement. It is built once, from real disagreements, signed off by whoever is accountable for that call — and nothing that comes after, including a future version of our own work, gets to grade itself instead of being graded by it.
A vendor's own number tells you how their tool does on the cases they chose. The plant's test set is the only number that survives the vendor leaving the room.
How it works, roughly
The test set a plant keeps after an engagement holds:
- The cases two experienced people on the floor have genuinely disagreed about, not the routine ones
- A written answer for each one, agreed and signed by whoever is accountable for that call
- A record of how the system in place at the time scored against it, as the starting line
When a new vendor, a new model, or a new idea from us turns up, it is run against that same set before anything else is discussed. The new score sits next to the old baseline, case by case, so a disagreement shows up on its own rather than hiding inside an average. Where an answer differs from the signed one, the difference goes back to whoever owns the sign-off — the person who can change the test set if the world has genuinely moved on, and the only person who can.
A year on, the set has usually outlived one piece of software it was built to judge, and it will outlive the next one too. What stays the same is who decides what counts as right, and where that decision is kept — inside the plant's own building, not whichever company happens to be selling this year.
A person still decides what a right answer looks like, on the cases nobody disputes are hard. The record of that decision, and of every test run against it since, stays in the customer's own building — the one part of the engagement built to outlast us.
See how we build the test set first →