A vendor evaluation, three systems in, settled into the rhythm of familiar demos. Then the third engineer taps a key and the line itself stops — not a screen showing what it would recommend, the physical line, mid-cycle, with no light asking anyone to confirm. He calls it full autonomy, and says it like the headline feature. For a second the room is impressed.
Then the plant head asks the question that should have come first: if it had been wrong, who would have known before the batch was already scrapped. The engineer says the system logs everything. That is not the same question, and everyone can feel it is not, even unsaid. What should annoy them is not that a shutdown might be wrong — a wrong shutdown gets fixed. It is that a decision nobody made cannot be explained, defended, or learned from. The line stopped, and there was no person left to ask why.
Why we build it this way
Automation is not the mistake here. A system that watches a hundred readings at once and never gets tired is exactly what a plant needs. The mistake is treating the moment a recommendation turns into an action as one more step to optimise away. That moment is where the cost lives — the batch scrapped, the order told no, the hour the line stands idle. Whoever carries that cost afterward should be the one who chose it.
The obvious answer, the one every vendor pitches, is that removing the person makes the system faster and the company more scalable. True in the narrow sense, wrong in the sense that matters. A system acting alone does not get more trustworthy with time — it gets less legible, because the record of why it acted stops being a decision and starts being a log entry nobody reads until after something has gone wrong.
The orange element is not a limitation we have not yet engineered away. It is the point of the exercise.
So the line we draw is specific, not sentimental. Anything that reads, measures, cross-references and proposes can run as fast and as constantly as the hardware allows. Anything that changes something physical, commits money, or closes an option the company cannot easily reopen waits for a person, every time, however confident the recommendation sounds. Confidence is not the same thing as accountability, and only one of the two can sign for what happens next.
How it works, roughly
The boundary is drawn the same way in every product we build, whether the decision is a diagnosis on a shopfloor or a stage change in a pipeline:
- What it reads: the readings or records in front of it, plus the history of similar decisions the plant or the business has made before
- What it gives back: a recommendation and the reasoning behind it, in the words a person on the floor would use — never a silent action taken on its own
- What happens when a person disagrees: the disagreement is recorded with the same weight as an agreement, and is often worth more, because it marks a case the system had not seen before
That last point does the most work. Confirming a recommendation teaches the system something. Overriding one teaches it more, because it marks the exact edge of what the system could not have known by itself — a call from a customer that changed the plan, a batch that reads wrong on paper but is fine, a quirk of one machine that never made it into a manual. Skipping the person does not just remove a check. It throws away the one case worth remembering.
A year on, none of the decisions that mattered were made by the system alone, and every one of them left a record of who made it and why. The wrong turns are in that record too, not edited out, because a company that owns its own history owns the near misses along with the confirmations. The orange element in every structure we draw is where that record comes from, not decoration on top of it: a person decides, and the record stays in the customer's building.
Read the vision →