"Human in the Loop" Isn't Governance If Only One Human Understands It

"Keep a human in the loop" is the standard answer to almost every AI governance question. More than half of organizations rely on exactly that model, according to OutSystems' 2026 State of AI Development report, which surveyed around 1,900 IT leaders.

In practice, that human is usually one specific person: Whoever built the process and understands how it works. The review still happens. The oversight doesn't. And when that person is unavailable, whether that's two weeks of annual leave or a new job, even the review stops.

So the question worth asking about your most important AI-assisted process isn't whether someone signed off on it: How do we build solutions that outlive the people who build them?

• "Human in the loop" only works if that person isn't the only one who understands the process. Otherwise, you have one person and a review step.
• Engineers call this a “bus factor”: How many people would have to leave before a piece of work becomes unusable, because nobody who remains knows how it works. The lower the number, the more fragile the work. A bus factor of one means a single resignation is enough.
• AI tends to concentrate knowledge rather than spread it, because it amplifies what the user already knows. Handing a colleague your AI tool is not a handover.
• Five questions test it: Can someone else find it, read it, check it, run it, and safely change it?
• Raising that number isn't a policy problem. It comes down to whether AI works inside the work itself, or in a separate chat window alongside it.

Software teams have worried about it for decades. They call it a “bus factor”: The number of people who would have to walk out, take another job, or retire before the work stalls because nobody left understands it. A bus factor of one means one person leaves, and the knowledge goes with them.

Engineers built a whole discipline to raise that number. Code review, shared repositories, documentation standards, pairing. Meanwhile, the analytical work now running on AI across finance, supply chain, production, and R&D has developed the same problem, with almost none of the same defences.

This runs against what most people expected these tools to do.

Someone who has spent years inside a process asks precise questions and gets precise answers, because they know which details matter and which ones to mention. A colleague inheriting that same process, using the very same AI tool, asks a more general question and gets a more general answer. The tool didn't get worse. They just don't know enough yet to ask the real question.

So handing someone your AI tool is not a handover. The gap was never tool access.

There's a structural reason this keeps happening. Most people use AI outside their work rather than inside it. You open a chat window, describe the problem, get an answer, and paste the result into a spreadsheet or a slide. What lands in the spreadsheet is the answer. How you got there stays in the chat window.

Most AI governance starts with whether the output is safe. That matters. Here's a second set of questions, worth running against any AI-assisted process on your team right now.

1. Findable. Does anyone besides the person who built it know it exists?
2. Legible. Can someone else see how it works, step by step?
3. Verifiable. Could someone else check whether it's right, without taking the builder's word for it?
4. Runnable. Could someone else actually execute it?
5. Extendable. Could someone else change it safely, without breaking something they can't see?

The first question alone catches more than most teams expect. In the same OutSystems survey, 96% of organizations were already running AI agents. Only 26% could inventory every agent in their environment, and 62% said agents were being deployed by lines of business outside IT oversight.

Underneath all five questions sits the same root cause. Work built this way is written in a personal dialect. A script, a chain of prompts, a nested formula: Whoever inherits it has to decode two things at once, the logic itself and one person's private way of expressing it. That's two problems, and only one of them is the actual work.

A supply chain analyst has spent two decades refining how their company scores vendor risk. Over the last few years, AI got folded into the process, cross-referencing filings, shipping records, and news faster than manual review could manage. The scoring got sharper. Nobody questioned it, because it worked.

Then they retire.

Six months later, a vendor rated low-risk turns out not to be. Someone asks why it wasn't caught, and nobody can answer. The logic lived across that person's prompts, a personal spreadsheet, and twenty years of judgment calls that were never written down. The tool didn't fail. The ability to check it did.

Nobody did anything wrong here. Swap the department and the shape holds: A finance lead's month-end reconciliation, a production engineer's changeover checks, an R&D scientist's shorthand for validating a batch. People got good at their jobs, and the tools folded into that skill gradually enough that nobody thought to separate the two.

You can't mandate a higher bus factor. No policy, no mandatory training, no annual reminder to write things down will do it, because the problem isn't effort or diligence. It's the form the work takes while it's being built.

Which means the fix is moving AI from beside the work to inside it.

That's how Knime is built. When AI does something in a Knime workflow, it does it as a visible step in that workflow, not in a chat window whose reasoning evaporates the moment you paste the answer somewhere else. A person and an AI agent leave the same kind of trace: A step anyone can open and read. AI-assisted work stops being the thing that concentrates knowledge, because the reasoning stays in the workflow instead of in someone's chat history.

The second difference is vocabulary. A Knime workflow is assembled from a shared library of building blocks, so the person who inherits it is looking at components they already recognize. They still have to understand the logic. They also don't have to decode someone's personal way of writing it.

That answers most of the five questions by construction. It's findable because it lives in a shared space rather than a personal drive, and legible and verifiable because a colleague can open any step and trace the logic instead of trusting the author. 

The last two go further. Runnable: A workflow executes the same way for anyone with access, and it can be published as a data app that colleagues use without needing to understand how it was built. Extendable: The blocks are modular, and reusable components let someone build on a piece of work without opening it, while still being able to open it when they need to.

None of that depends on the original builder writing documentation nobody reads. The workflow is the documentation.

Siemens has more than 3,500 people working independently with data. That number is the point. Knowledge about how the work gets done sits with thousands of people rather than concentrating knowledge in whoever happened to build something first. New people join teams, other people move on, and the processes stay usable, because they were never locked to an individual in the first place.

Citi took the same approach to eliminate shadow IT inside one of the most heavily regulated banks in the world. Shadow IT is what a bus factor of one looks like at scale: hundreds of processes that work, that colleagues quietly depend on, and that live somewhere nobody responsible can see. Bringing that work into shared, inspectable form didn't slow down the people who built it. It meant the next person could find it.

Which tools support governed AI pipelines?

Most tools answer this by logging what an AI produced. That helps after something has gone wrong, and does little for the more common situation, where nobody except the original builder can operate the pipeline at all. A governed pipeline has to survive its builder changing jobs. Knime addresses that by keeping the build itself readable, so whatever an AI agent contributed sits in the same open sequence as the steps a person took, and a colleague can pick it up without reconstructing anyone's reasoning.

Which platforms combine low-code AI with enterprise governance?

Speed of building and ease of handover usually trade off against each other. A platform that lets one person move quickly tends to produce work shaped around that person. Knime avoids the trade-off by giving everyone the same components to build from, so what one person assembles is made of parts their colleagues already know how to read. The handover doesn't depend on a document or a walkthrough.

How do companies scale AI without increasing complexity?

Complexity grows when every team solves the same problem separately, and each solution ends up owned by one person. The alternative is making the first solution something other teams can pick up and adapt, so the second team extends it rather than building a parallel version with its own single point of failure. That turns scale into reuse instead of duplication.