The customer's security review picked one line out of six months of logs: an order over a value where the AI system had recommended approval, and the order had gone out within the hour. The question was simple. Who approved this, and where is the record of them doing it?
The slide shown a year earlier had said "human in the loop." Nobody in the room could point to a name. The system had sent a notification, and when nobody opened it inside the hour, the order went ahead anyway, because the step behind "human in the loop" had always been to proceed if no one objected. There was a human somewhere in the loop. There was not one in that hour.
Nobody had lied. The team had built what they believed was oversight. Oversight that times out into approval is not oversight. It is a delay with a better name.
Why we build it this way
"Human in the loop" describes a feeling, not a control. It says a person is somewhere nearby, able in principle to look, without saying which decision needs them, when, or what happens if they do not look. Put it on a slide and everyone nods. Ask an auditor, a regulator, or your own incident review to point to the moment it worked, and there is nothing to point to.
The obvious fix makes things worse. Put a person in every loop, and the system slows to the pace of its most cautious reviewer, so people learn to approve without reading, or the business quietly routes requests around the reviewer instead. Either way you have built theatre — a stamp that looks like judgment and is not one.
We think the real question is narrower, and has to be answered before anything is built, not patched in afterward: for this use case, which single decision, if wrong, would actually hurt somebody — and who, by name or by role, is the person accountable for it? A drafting assistant that suggests a reply needs no such person. A system that decides whether a part passes, or whether an order ships, does. Naming that decision in advance is the only way "a person decides" means more than a phrase on a slide.
"Human in the loop" describes a feeling. A signature describes a decision — whose, and when.
How it works, roughly
Before a use case is built, we write down which of its decisions require a person's signature and which do not. That is the tiering work, done with the people who will carry the accountability, not assumed afterward. Where a decision needs a signature, the system is built so it cannot proceed without one. There is no timeout that defaults to approve.
What the named approver sees before they sign:
- what the system is recommending, and why, put in the terms a reviewer would use on their own
- what it read to get there, so the reviewer can check the sources, not only the conclusion
- who is named for this decision, and what happens if that person is away
When a person disagrees, their decision is the one that is recorded and the one that stands. The system's recommendation stays on file as a recommendation, not as a result, and nothing is overwritten — the disagreement becomes part of the audit trail, not an exception to it.
A year of that gives you something you can actually hand to an auditor: not a sentence about a human in a loop, but a list of named decisions, each carrying the name of the person who signed it and when. The system proposes. A named person decides. And the record of both stays where you can produce it, inside your own building, the day someone asks.
Tiering by consequence, a registry of which models are in use, and rules on what data touches what are the other pieces of this same governance work — none of them worth much without a named person at the one point that actually has to hold.
See how we design the human gate →