The score you never checked again

Three vendors sat in a room six months ago and answered the same twenty questions your best engineer could think of. One of them answered best, so you signed, and the number from that afternoon went into the slide that got the deal approved.

The system has been running since, quietly, on real orders and real drawings. Nobody has asked it those twenty questions again.

In the meantime a new supplier came onto the approved list. The engineer who wrote the questions moved to a different plant. The vendor pushed an update over the weekend, the kind the contract does not require them to tell you about. If the system is answering worse than it did on the day you signed, the first sign of it will be a person downstream catching a wrong answer, not a report that told you first.

Why we build it this way

Most organisations treat the score from the sales process as a permanent fact about the system. It is not. It is a photograph of one afternoon, taken by the party trying to win the business. Signing off on that photograph and never taking another one is not caution. It is the appearance of caution.

The deeper problem is who owns the test. A benchmark the vendor wrote is the wrong instrument for checking the vendor's own work later, the same reason a supplier should not grade its own incoming inspection. And a test built once, for one bake-off, has no reason to still describe the work a year on — the parts change, the failure modes change, the person who understood the edge cases may not even be in the room any more.

So we build the test set with the people who actually argue about the hard cases, not with the vendor who is selling into the room. It stays yours after we leave, in a form your own team can run again without calling anyone.

How it works, roughly

The set itself reads like the cases your engineers already disagree about:

• The ones argued over in review, where two experienced people gave different answers
• The edge cases that only came up once, but mattered when they did
• A baseline: how the current system, or the one you are replacing, actually performs against them today

Run again, it gives back a comparison against the last run, not a single new score — where it improved, where it slipped, and a catalogue of the ways it still gets things wrong, so a new mistake reads as a known pattern rather than a surprise. When the vendor changes something underneath, or a person wants a second opinion before renewing a contract, this is what runs before that decision, not after it.

When a person disagrees with the result, the comparison does not ship anything on its own. It goes to whoever owns the decision to accept the change, hold it back, or roll it over to the previous version — the same person who signed off on the system in the first place.

AI Transformation · drag to turn