The 95% trap

When AI output was rough, we had no choice but to engage with it. We rewrote, restructured, checked. Now that it’s good — sometimes really good — something counterintuitive is happening. The better it gets, the less we check. And the errors that slip through are often the ones that actually matter.

This is not a new observation. Automation researchers have studied similar dynamics for decades under names like automation bias and automation complacency. Pilots miss instrument errors because autopilot usually works. Doctors adopt incorrect machine suggestions because decision-support systems are usually right. The pattern is well established: the more reliable a system appears, the less carefully humans monitor it.

And yet, most AI workflows in knowledge work follow the same pattern: prompt, generate, review, skim, use. We’ve put all the human judgement at the end. And the end is exactly where humans are worst at applying it.

(this follows on from my recent piece about AI as transitional scaffolding, and how today’s productivity gains might be masking deeper problems – this is one of those problems)

Why knowledge work changes the problem

In aviation or clinical diagnosis, the AI produces a discrete output: an alert, a recommendation, a classification. You can check it against other information. In knowledge work, AI produces something harder to verify: plausible prose.

A research synthesis, a policy summary, a strategy document. These are not right or wrong in the way a medical diagnosis is. They’re persuasive or unpersuasive, complete or incomplete, well-framed or subtly distorted. The errors AI makes in this kind of work are not obvious factual mistakes. They’re smoothed tensions, missing caveats, distorted emphasis, shallow synthesis dressed up as insight. Catching those requires sustained engagement, not a quick scan.

It’s exacerbated by a decades-old human failure that makes the quick scan so tempting. We tend to use presentation quality as a proxy for truthfulness. We’ve always done this – subconsciously a lot of the time. A well-formatted, confidently written document feels more credible than rough notes, even when the content quality is identical. Psychologists call this the fluency heuristic. AI outputs maximise every signal that triggers it: grammatically perfect, well-structured, rhetorically confident. The result is text that actively discourages you from questioning it.

The issue is not just that AI is wrong sometimes. It is that it is wrong in ways that seem finished.

A different workflow architecture

Most discussions of this problem land on the same advice: be more careful when you review AI outputs. That is perhaps the least useful advice, because it is asking us to do exactly the thing that research shows we are inherently bad at.

The response needs to be structural instead. There’s a distinction I keep coming back to: human in the loop versus human *guiding* the loop. They sound similar but describe fundamentally different relationships.

Human in the loop means the AI acts and the human approves. It’s a checkpoint at the end of the process, and it’s the model most people use now. Human guiding the loop means the human shapes the AI’s reasoning throughout: framing the problem before generation, steering intermediate outputs, injecting domain knowledge at multiple points. The AI’s reasoning unfolds under continuous direction, not in a single generation-then-review cycle.

Why this might help

1. You spend most of your time working with rough artefacts. Outlines, bullet fragments, partial syntheses, AI reasoning notes. These not only are unfinished but they look unfinished. They don’t trigger the fluency heuristic the way a polished final draft does. Rough outputs create psychological permission to critique in a way that polished outputs actively suppress.
2. You catch framing errors early. A wrong assumption in paragraph one of a final draft has already shaped everything that follows. The same wrong assumption in an initial outline is easy to spot and correct before it propagates. The cognitive load of challenging one early decision is far lower than unpicking a finished synthesis.
3. You stay cognitively engaged. The automation research is clear on this: humans are worse monitors when they’re passive. When you’re actively participating in shaping the output, you maintain a mental model of what it’s trying to do and why. That makes you a better judge of whether it’s actually doing it.

And in development and policy contexts especially, guided workflows create explicit entry points for contextual knowledge. A practitioner in the Global South working with a guided process has structural moments to inject local knowledge, correct assumptions, challenge framings that reflect Northern institutional consensus.

Traditional generate-then-review workflows don’t create those moments. The output arrives fully formed, carrying whatever biases its training data contained, and the review step is where you’re supposed to catch all of that. Under time pressure, with a document that looks authoritative . . . that is hard!

What this doesn’t solve

I don’t want to oversell this.

Human-guided workflows improve the conditions for judgement but they don’t guarantee it, and they introduce their own risks.

The most obvious one is confirmation bias. When you co-produce something with AI, steering it at every stage, the result aligns closely with your thinking. It feels like yours. That sense of ownership may actually make you less likely to challenge it, not more. Confirmation bias is not new (let’s be honest, people were cherry-picking evidence and writing toward preferred conclusions long before AI arrived) but AI makes the process of building a well-rationalised, fluent, persuasive case for whatever you already believed faster, cheaper, and more convincing-looking than ever.

Then there’s epistemic outsourcing. Even in a guided workflow, the AI is still doing substantial synthesis work. If you lack the subject depth to challenge what it produces, guiding the process may just give you a stronger feeling of understanding without the actual understanding to back it up.

And the power dynamics don’t disappear. AI can lower the cost of finding alternative perspectives, surfacing critiques, mapping whose voices are missing from a body of evidence. But it only does that if the workflow explicitly demands it. A Northern researcher using a guided workflow will still get Northern-dominant outputs unless they deliberately design prompts that challenge that. The workflow creates the opportunity; it doesn’t guarantee the outcome.

Some questions I can’t answer yet

A few things genuinely bother me about this argument, and I think they’re worth naming rather than glossing over.

• Does distributing judgement through the process actually reduce AI errors, or does it just move them earlier where they’re easier to notice?
• Does guiding the AI engagement genuinely build a better understanding of the material, or does it just create a stronger illusion of understanding? The feeling of ownership is definitely real, but does it correspond to actual epistemic knowledge?
• Can better workflow design meaningfully shift whose knowledge gets centred, or does it mainly help people who are already well-positioned to steer AI effectively? I’d like to think it does, but if the answer is no, then human-guided loops are a useful improvement for privileged knowledge workers, but maybe not much else.

( I’ve been working on a structured approach to implementing this type of workflow (which I already use in my AI-assisted blogging) – I’ve been calling it the EFFECT workflow: Explore, Freeze, Frame, Execute, Critique, Test. More on that in a future post! )

I suspect the honest answer is that this is a better model than what most people are currently doing, but it is not sufficient on its own. Some tasks may simply be too verification-intensive for AI-assisted speed to be worth it. For the rest, the question is not “should we use AI?” but “how do we design the process so that human judgement remains central and steering rather than decorative at the end?”

That is a workflow design question, not a willpower question. And it’s one I think we need to take more seriously.