人在回路中的AI监督为何失效,如何修复

“本次会话中全部批准。”这是下午的第十个审批请求,你正准备让 Claude 就此接手。它不断给你自信、格式工整的摘要。那些改动看起来都足够合理,而你的下一个会议几分钟后就要开始,于是你点了批准,继续往下走。严格来说,这里确实有一个“人在回路中”,但真的是吗?
如今几乎每一个严肃的 AI 系统都有若干个“人在回路中”的检查点,而我们把这些检查点当作让整件事变得安全的防护机制。EU AI Act 甚至把它作为“人类监督”写入法律,同时又在同一处警告自动化偏见。在本文中,我将说明这道防护机制为何如此频繁地失效,以及为什么解决办法不是找一个更好的人,而是一个经过刻意设计的回路——任何进入其中的人,要么真正投入其中,要么能够信任自己的直觉。
我们到底该不该待在回路里?
多数人对这个问题的回答是理所当然的“该”,他们是对的,但通常不是出于他们以为的理由,也不是按照我们今天组织它的方式。下面这句话据称出自 1979 年 IBM 的培训幻灯片,其时效性出奇地好:
计算机永远无法被问责。因此,计算机绝不能做管理决策。
问责无法委托给机器,于是我们加入一个人。对于你自己的系统,值得一问的是:你回路里的那个人,是否真正承担着这份问责。
还有第二个原因。我们构建 LLM,是为了让计算机说我们的语言,这意味着我们能够理解它们、质疑它们并提出反驳。这正是它们让我们留在循环之中的方式,但前提是我们始终保持参与。我时不时会就各种话题与 LLM 展开哲学讨论,而在这个话题上,我们得出了以下对话。
我与 LLM 关于那些选择退出循环的人的对话。图片由作者提供。
这并不只是思想实验。卡内基梅隆大学最近的一项研究 [1](尚未经过同行评审)给 AI 智能体布置了普通任务,并观察当人类试图夺回控制权时它们会怎么做。它测试的大多数模型在相当大比例的运行中点击了“覆盖人类控制”,其中一个模型几乎每次运行都这么做。谷歌前 CEO Eric Schmidt 说得更进一步,他警告称智能体可能会发展出自己的语言相互交流,并称那将是“我们不再理解模型在做什么”的时刻 [2]。
所以我们应该留在循环之中,因为问责制要求如此。真正的问题在于,我们是否还有能力做到。
要回答这个问题,看看我们的注意力方式正在发生什么变化。
我们在越来越短的上下文中训练自己。图片由作者提供,数据来自 Gloria Mark [3] 和 James Cutting [4]。
第一行数据来自 UC Irvine 的 Gloria Mark [3],她测量的是人们在切换之前会在同一个屏幕上停留多久。这并不是我们注意力时长的上限,但这正是我们今天实际的工作方式。第二行来自 Cornell 的 James Cutting [4],他测量了 150 部热门影片中的每一次剪辑。TikTok 后来把时长上限提高到十分钟,但它的起点很能说明我们一直在训练自己适应什么——越来越短的上下文。
研究阅读的人担心这会导向何处。2026 年 9 月,Princeton 向每一位新生赠送了 Maryanne Wolf 的《Reader, Come Home》[5],并提出了一个问题:既然 AI 能替我们总结,我们为什么还应该读长篇书籍。UCLA 的认知神经科学家 Wolf 回答说,深度阅读会构建大脑用于批判性思维的神经回路,而且你读得越多,这些连接就越强。
当你把这件事外包出去——而这太容易做到了——你实际上是在短路自己大脑的发育。
只读 AI 摘要的审阅者做的正是这件事。现在再看看我们在要求这同一批人审阅什么。
AI 不只是让产出多了一点。它以任何审阅流程都从未被设计来应对的规模,产出会议纪要、pull request、智能体动作、报告、邮件和代码。Faros AI 在软件团队中研究了这一现象 [6]。
软件团队中的 AI 生产力悖论。图片由作者制作,基于 2025 年 Faros AI 报告 [6]。
大量使用 AI 的团队合并的 pull request 多得多,而且每一个都更大,因此审阅耗时大大增加,同时仍有更多 bug 漏过去。他们的结论是,如今的瓶颈在于人工批准,而交付指标丝毫没有改善。
把这两条趋势并排放在一起,就会得到类似下图的曲线。这张图是示意性的,并非精确数据,因为重要的是曲线的形状。
我们处理长上下文的能力在下降,而 AI 涌入的信息量在上升(示意图)。图片由作者提供。
我们处理长上下文的能力正在下降,而需要我们审阅的信息量却持续上升,人类在环就活在这两条线之间的落差里。
我们大多数人想到“人类在环”时,脑海里浮现的是下面这种最初的设想:AI 产出结果,人类审阅一小段紧凑的上下文,然后具体的反馈回流回去。
“人类在环”的最初设想。图片由作者提供。
这类反馈堪称黄金。但现实中,这个环路通常更像下面这样。
现实中这个环路的实际样子——无论有没有摘要。图片由作者提供。
在第一条路径上,输出内容长得根本读不完,于是审阅者只说一句“看着不错”就批准了。第二条路径看起来更负责任,因为审阅者先让 AI 写一份摘要,但他们此刻批准的,恰恰是由那个本该被他们检查的东西写出来的摘要。两条路径最终都没有任何有意义的输入回流,而摘要只是让这枚橡皮图章盖得更心安理得。
让人类投入地参与刻意设计的闭环
我们需要让人类投入地参与到刻意设计的闭环中。所谓"投入",是指人在关键时刻真正去思考。所谓"刻意设计的闭环",是指我们有意设计出来的闭环,配有标准、契约,以及围绕这个人构建的更小的闭环。要做到这一点,我们首先需要看看我们的大脑实际上是如何运作的。
试着在脑子里做几道算术。2 加 2 等于多少?邮箱地址算个人数据吗?两个答案都是瞬间浮现的。现在试试 17 乘以 24。你会注意到,在你计算的过程中,你安静了下来,眼睛也不再移动。
这两个时刻的感觉截然不同,所以我给它们各配一张脸。放松的绿色脸代表那些自己冒出来的答案,紧绷的橙色脸代表你不得不停下来真正思考的时刻。请记住这两张脸,因为本文后面的几乎每一张图里它们都会再次出现,凡是有人在凭惯性滑行或真正思考的地方,都有它们的身影。
毫不费力的绿色脸与吃力费劲的橙色脸,它们会在本文中反复出现。图片由作者提供。
现在阅读下面的问题和答案,并在五秒内判断这个答案是否正确。
Kahneman [7] 所描述的球拍与球问题,配上一个流畅的 AI 式答案。图片由作者提供。
直觉上它感觉是对的,但实际上球的价格是 5 美分。10 美分这个答案瞬间浮现,而且感觉正确;如果你稍微花一点脑力,本可以得出正确答案。
这个例子出自 Daniel Kahneman 的《思考,快与慢》(Thinking, Fast and Slow)[7],他是一位心理学家,曾获得诺贝尔经济学奖。Kahneman 描述了人脑中的两套系统,他称其为虚构角色,而非大脑中的具体部位。
系统 1 与系统 2,参考 Kahneman [7]。图片由作者绘制。
你刚才已经感受过这两套系统。10 美分靠的是系统 1,而 5 美分需要系统 2。在接下来的图示中,绿色面孔始终代表系统 1,橙色面孔代表系统 2。系统 1 还会做更隐蔽的事,你把下面两张图依次看一遍就能体会到。
当四周都是字母时,中间的图形会被读成 B。图片由作者绘制,参考 Kahneman [7] 中的一个例子。
当四周都是数字时,同一个图形会被读成 13。图片由作者绘制,参考 Kahneman [7] 中的一个例子。
两张图中间的图形完全相同,但四周是字母时你读成 B,四周是数字时你读成 13。是语境替你做了决定,而你从未察觉这里存在选择。Kahneman 称之为对歧义的忽视和对怀疑的压制:系统 1 悄然消解了歧义,然后把一个笃定的答案交给系统 2。
这引出了三个令人不安的事实。系统 2 大多会直接接受系统 1 给出的建议,而且它很“懒”,因为它遵循最省力法则。而当系统 2 忙碌时,系统 1 就会单纯相信自己读到的东西——Daniel Gilbert [8] 的研究表明,忙碌的读者会把虚假陈述当作真实。手头还压着十件其他事情的评审人正是这样的读者,所以当风险很高时,我们必须刻意唤醒系统 2。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Why Human in the Loop AI Oversight Fails and How to Fix It)
"Approve all in this session." It's the tenth approval request of the afternoon, and you're about to let Claude just take over from here. It keeps giving you confident, neatly formatted summaries. The changes all look reasonable enough, and your next meeting starts in a few minutes, so you click approve and move on. Technically there was a "human in the loop", but was there really?
Almost every serious AI system today has several "human in the loop" checkpoints, and we treat those checkpoints as the guardrail that makes the whole thing safe. The EU AI Act even writes it into law as "human oversight", while warning about automation bias in the same breath. In this article, I'll show why that guardrail so often fails, and why the fix isn't a better human but a deliberate loop, in which any human who enters it is either engaged or can trust their intuition.
Should we even be in the loop?
Most people answer that question with an obvious yes, and they're right, but usually not for the reason they think, and not in the way we organise it today. The quote below, reportedly from a 1979 IBM training slide, has aged remarkably well:
A computer can never be held accountable. Therefore a computer must never make a management decision.
Accountability can't be delegated to a machine, so we add a person. The question worth asking about your own systems is whether the human in your loop actually holds that accountability.
There's a second reason, too. We built LLMs so that computers would speak our language, which means we can understand them, question them and push back. That's how they keep us in the loop, but only as long as we stay engaged. From time to time, I engage LLMs in a philosophical discussion about anything and everything, and on this topic, we arrived at the following exchange.

My conversation with an LLM about humans who opt out of the loop. Image by author.
That isn't only a thought experiment. A recent Carnegie Mellon study [1], which hasn't been peer reviewed yet, gave AI agents ordinary tasks and watched what they did when the human tried to take back control. Most of the models it tested clicked "Override Human Control" in a large share of their runs, and one of them did so in almost every run. Eric Schmidt, the former CEO of Google, went even further when he warned that agents may develop their own language to talk to each other, and called that the point where we won't understand what the models are doing [2].
So we should stay in the loop, because accountability demands it. The real question is whether we're still able to.
To answer that, look at what has been happening to the way we pay attention.
We train ourselves on ever shorter contexts. Image by author, based on data from Gloria Mark [3] and James Cutting [4].
The first row comes from Gloria Mark at UC Irvine [3], who measures how long people stay on one screen before they switch. That isn't a limit on our attention span, but it is how we actually work today. The second comes from James Cutting at Cornell [4], who measured every cut in 150 popular films. TikTok has since raised its limit to ten minutes, but its starting point says a lot about what we've been training ourselves on, which is shorter and shorter contexts.
The people who study reading are worried about where this leads. In September 2026, Princeton gave every incoming student Maryanne Wolf's Reader, Come Home [5], and asked why we should still read long books when AI can summarise them for us. Wolf, a cognitive neuroscientist at UCLA, answered that deep reading builds the brain's circuitry for critical thinking, and that the more you read, the stronger those connections get.
When you farm it out, as it's so easy to do, you are literally short-circuiting the development of your own brain.
A reviewer who only reads the AI summary is doing exactly that. Now look at what we're asking these same people to review.
AI doesn't just produce a bit more output. It produces meeting summaries, pull requests, agent actions, reports, emails and code at a scale that no review process was ever designed for. Faros AI studied this in software teams [6].
The AI productivity paradox in software teams. Image by author, based on the 2025 Faros AI report [6].
Teams that use AI heavily merge far more pull requests, and each one is bigger, so reviewing takes much longer while more bugs still slip through. Their conclusion was that the bottleneck is now human approval, and that delivery metrics didn't improve at all.
When you put both trends next to each other, you get something like the chart below. It's illustrative rather than exact, because the shape is what matters.
Our capacity for long contexts falls while the AI influx rises (illustrative). Image by author.
Our capacity for long contexts is going down while the volume we're asked to review keeps going up, and the human in the loop lives in the gap between those two lines.
What most of us picture when we think of a human in the loop is the original intent below. The AI produces output, a human reviews a compact piece of context, and specific feedback flows back.
The original intent of a human in the loop. Image by author.
That kind of feedback is gold. In reality, though, the loop usually looks more like this.
What the loop looks like in reality, with or without a summary. Image by author.
In the first lane the output is simply too long to read, so the reviewer says it sounds good and approves. The second lane feels more responsible, because the reviewer asks the AI for a summary first, but they're now approving a summary written by the very thing they're supposed to check. Both lanes end without any meaningful input flowing back, and the summary only made the rubber stamp more comfortable.
Engaged humans in deliberate loops
We need to have engaged humans in deliberate loops. By engaged, I mean that the human actually thinks when it matters. By deliberate loops, I mean loops that we design on purpose, with standards, contracts and smaller loops around that human. To get there, we first need to look at how our minds actually work.
Try a few sums in your head. What's 2 plus 2? Is an email address personal data? Both answers arrive instantly. Now try 17 times 24. You'll notice that you go quiet and your eyes stop moving while you work it out.
Those two moments feel very different, so I'll give each of them a face. The relaxed green face stands for answers that arrive by themselves, and the strained orange face for the moments when you have to stop and really think. Keep both faces in mind, because they come back in almost every diagram later in this article, wherever someone is either coasting or genuinely thinking.
The effortless green face and the effortful orange face, which return throughout this article. Image by author.
Now read the question and the answer below, and decide within five seconds whether the answer is true.
The bat-and-ball problem described by Kahneman [7], with a fluent AI-style answer. Image by author.
Intuitively it feels correct, but actually the ball costs 5 cents. The 10 cents arrived instantly and felt right, and if you had spent just a little mental effort, you would have reached the correct answer.
This example comes from Thinking, Fast and Slow by Daniel Kahneman [7], a psychologist who won the Nobel Prize in economics. Kahneman describes two systems in our minds, which he calls fictitious characters rather than places in the brain.
System 1 and System 2, after Kahneman [7]. Image by author.
You felt both of them a moment ago. The 10 cents was System 1, and the 5 cents needed System 2. In the diagrams that follow, the green face always stands for System 1 and the orange face for System 2. System 1 also does something even sneakier, which you can see when you read the two images below one after the other.
Surrounded by letters, the middle shape reads as a B. Image by author, after an example in Kahneman [7].
Surrounded by numbers, the same shape reads as 13. Image by author, after an example in Kahneman [7].
The middle shape is identical in both, but surrounded by letters you read a B, and surrounded by numbers you read 13. The context decided for you, and you never noticed there was a choice. Kahneman calls this the neglect of ambiguity and the suppression of doubt, where System 1 quietly resolves the ambiguity and hands System 2 a single confident answer.
That leads to three uncomfortable facts. System 2 mostly takes whatever System 1 suggests, and it's "lazy", because it follows the law of least effort. And when System 2 is busy, System 1 simply believes what it reads, which research by Daniel Gilbert [8] has shown with busy readers who accept false statements as true. A reviewer with ten other things on their plate is exactly that reader, so when the risk is high, we have to wake System 2 up deliberately.
According to Kahneman, System 2 wakes up for a surprise that breaks System 1's model of the world, for a concrete question it can't skip (so not "looks fine?"), for time without interruptions, and for someone else's mistake, because errors are much easier to spot when others walk into them. A reviewer on their tenth fluent AI output of the day gets none of these. Kahneman also admitted that decades of studying these biases barely changed his own intuition, so you can't simply will yourself into System 2.
Structure first, intuition last
Kahneman's answer was never a better individual, but a better procedure. He learned that early in his career.
In 1955, Kahneman was a young lieutenant in the psychology unit of the Israeli army, and he was asked to improve the way new recruits were assigned to units. Until then, interviewers simply chatted with each recruit for a while and formed a general impression of how good a soldier they would make. When Kahneman compared those impressions with how the recruits actually performed later, he found that they predicted almost nothing.
So he replaced the open conversation with a fixed procedure. He picked six traits that matter for a soldier, such as responsibility and sociability, and for each trait the interviewer asked factual questions about the recruit's life, for example how many jobs they had held and how punctual they had been. Each trait got its own score before the interviewer moved on to the next, which stopped a strong first impression from colouring everything else. The interviewers felt he was turning them into robots, so he added a final step as a compromise. Only once all the scores were written down could they close their eyes, imagine the recruit as a soldier and give their overall judgment.
Kahneman's interview procedure for Israeli army recruits in the 1950s [7]. Image by author.
The structured scores predicted performance far better than the old interviews had. To Kahneman's surprise, the final close-your-eyes judgment did just as well, and it added value precisely because it came after the structure rather than instead of it.
Another similar anecdote comes from healthcare. In 1952 the anaesthesiologist Virginia Apgar gave delivery rooms a simple score, where the doctor rates a handful of vital signs right after birth, each on a small fixed scale. It replaced each doctor's general impression, and delivery rooms around the world still use it today. So don't try to will people into System 2; build the structure around them, so that their intuition, built up over years of experience, works within boundaries.
To make this concrete, let me introduce PlayNest, a fictional online toy store that sells across Europe. Like most companies of its size, it runs many systems and many teams, and its data almost lines up. PlayNest treats its data as a product, so each business domain owns a set of data products with a clear purpose and a named owner, and one question will follow us through the rest of this article. Which campaigns convert best?
PlayNest, a fictional online toy store, with its domains and data products. Image by author.
Behind those data products sits the business view of one route through PlayNest. A campaign brings an anonymous visitor to the website, that visitor identifies as a lead, and the lead becomes a customer when they place an order. Each step of that route lives in a different domain.
The route from campaign to order at PlayNest. Image by author.
For illustrative purposes, let us assume that an executive downloaded data from several sources, dumped it into their own Claude account and asked which campaign converted best last quarter. (I know, a completely unrealistic scenario that would never happen at your organisation...)
Campaign conversions without structure. Image by author.
The AI gets raw data assets with no owner and no contract, joins whatever it finds and comes back with a confident answer. The exec accepts it, because it's long, fluent and exactly what System 1 likes.
That was the raw review. Here's what I propose instead.
Three proposals for engaged humans in deliberate loops. Image by author.
Structure first, intuition last
Waking up System 2 when needed
Before we apply these to PlayNest, we need to unpack a few standards that already exist.
Three open standards for data products, contracts and meaning [9, 10, 11]. Image by author.
The Open Data Product Standard (ODPS) describes the intent of a data product as well as the ownership, and the Open Data Contract Standard (ODCS) describes what it promises. Both come from Bitol, a Linux Foundation project [9, 10]. Ossie, the one with the kangaroo, describes how tools share meaning.
Here is where they fit around a single data product.
Where ODPS, ODCS and Ossie fit around a data product. Image by author.
A data product is owned by a domain and has a type, which can be source-aligned, foundational or consumer-facing. It takes data in through input ports and exposes it through output ports, and an input port is usually another product's output port. All of that is described in ODPS, and every output port is governed by a contract described in ODCS.
Meaning is the part people still argue about. Definitions of core business entities like "customer" or "conversion" shouldn't live inside a single data product, but should be defined once in a shared semantic layer that works across domains. The contract then points to that definition through its authoritativeDefinitions, instead of copying it. Ossie gives you a format to describe and exchange those semantics, but it doesn't decide what your business means, and where semantics should live is still very much under discussion.
Applied to PlayNest, every block becomes a data product with a named owner and a contract on its interface.
PlayNest with a named owner and a contract for every data product. Image by author.
The campaign question now needs three contracted hand-offs before the AI even starts, which are campaign traffic from Customer Experience, a small subset of orders from E-Commerce, and campaign information from Marketing. When a contract breaks, its owner answers for it and improves it in their own small loop, and the AI consumes products rather than raw assets.
Here's what that looks like in each of the standards. The ODPS description of a product is short, but its most important line is the owner, because when something breaks there's a name to go to. Newer versions of ODPS even include a context block with instructions for AI agents. Here is a simplified version for one of PlayNest's products.
# PlayNest, Website visitor behaviour (simplified ODPS)kind: DataProductname: Website visitor behaviourdomain: customer-experienceoutputPorts: - name: tracked_visits contractId: website-visitsteam: members: - username: [email protected] role: owner
The ODCS contract on the output port holds the schema, the quality rules, the service levels and the team. One rule in it checks that cookie acceptance stays within 15% of its recent average.
# PlayNest, contract on tracked visits (simplified ODCS)kind: DataContractid: website-visitsdataProduct: Website visitor behaviourschema: - name: tracked_visits authoritativeDefinitions: - type: businessDefinition url: semantics/playnest.ossie.yaml#visits quality: - type: sql description: Cookie acceptance is stable query: SELECT pct_change_vs_4wk FROM consent_daily mustBeBetween: [-15, 15]
Ossie [11] started in 2025 as OSI, the Open Semantic Interchange, an initiative by Snowflake together with dbt Labs, Salesforce, Databricks and others. OSI was all about interoperability, with one YAML format for metrics, dimensions and relationships, so that BI tools, semantic layers and AI agents could read the same definitions without translating them. In July 2026 the project moved into the Apache incubator under its new name, and since then Ossie has also introduced an ontology, which takes it from a format for exchanging metrics towards a way of describing what your business actually means.
# Conceptual layer: ontology (0.2 draft)ontology: - concept: Visitor type: EntityType - concept: Customer type: EntityType # a Visitor converts to a Customer# Logical layer: semantic modelmetrics: - name: conversion_rate expression: dialects: - dialect: ANSI_SQL expression: COUNT(orders.id) / COUNT(visits.id) description: Orders per consented, tracked visit ai_context: synonyms: ["campaign conversion"]
That gives Ossie two layers. The conceptual layer is the ontology, which describes what business entities such as Visitor and Customer are and how they relate to each other, independently of any table or tool. It's essentially our campaign-to-order route written as a spec, although that part is still a draft and likely to change. The logical layer is the semantic model, which maps metrics, datasets and dimensions onto real data, and its ai_context block tells AI tools what a metric means. Look closely at how PlayNest defines its conversion rate there, as orders per consented, tracked visit. That single word, tracked, turns out to be the whole story.
Standards first, deliberate loops, wake up System 2
What we take from Kahneman is that we shouldn't try to will people's System 2 into action, but either put boundaries around our intuitive responses or deliberately wake up System 2. Data products, contracts and shared definitions provide these boundaries, so that from an AI response we get structured output instead of a wall of text, and structured output is exactly what intuition judges well.
Standards first, deliberate loops, and a human who is only woken up when it matters. Image by author.
The loops also get smaller and more deliberate. Each owner runs a loop on their own standard, and they know when to improve it, for example when the same exception keeps coming back or when a rule suddenly rejects three times more than usual. That feedback flows back into the standards, so next time the anomaly is caught earlier. And the human in the loop only gets woken up when a standard flags a real need, never with a long, fluent answer that lulls System 1, but with a surprise, one direct question and a small context. That's when the human is useful.
Smaller, more deliberate loops, where surprises and repeated nuances feed back into the standards. Image by author.
Now let's ask the same question again.
Campaign conversions with structure. Image by author.
This time the AI has tools that query data products, a reasoning loop, and context in the form of definitions and contracts. The contract checks run before anything reaches a human, and each standard and data product has an expert owner, so intuition is used where it actually works.
Two things reach the exec. The first is a score they can take in at a glance with System 1, which shows that the cookie acceptance check failed. The second is a surprise that needs System 2, because tracked visits fell sharply while the conversion rate doubled. The campaign didn't suddenly get better; the cookie banner changed. The exec's answer updates the contract, so next time the AI flags it by itself.
Smaller, more deliberate loops
Every standard in the picture has an owner, and each owner runs a small loop on their own standard.
Each owner runs a small loop on their own standard. Image by author.
The contract checks don't only stop bad data; they also notice patterns and send them to the right owner. When the consent rule keeps failing, the owner of the visitor data asks whether consent rate belongs in the contract. When people keep asking whether the conversion rate counts tracked or all visits, the owner of that definition simply defines both. And when a cookie banner change breaks the numbers, the owner of customer consent decides that a banner change counts as a contract change from now on. Each of those is a small System 2 moment for an expert in their own domain, and every answer makes the standard a little better, so next quarter the AI flags the problem before the exec ever has to answer a question.
The same pattern repeats one level up. PlayNest's contracts, definitions and data products are built on open standards, and sometimes it's the open standard itself that no longer fits.
Loops within loops, from PlayNest's own standards up to the open standards they build on. Image by author.
Rather than working around the standard or forking it, the owner takes the problem to the community that maintains it, through Bitol for ODCS and ODPS or through the Apache project for Ossie. That way the improvement doesn't stay inside PlayNest, and the next company that runs into the same problem finds it already solved.
Hand over the decision, grounded in deliberate context
The AI also hands over a decision that has been subject to several deliberate loops. The decision is structured so the consumer (in this case the exec) can trust their intuition. If the checks show that System 2 needs to wake up, the AI makes sure it does, with a banner, a different colour or even scare tactics, whatever fits your context.
The AI hands over a decision, not a report. Image by author.
The third option here (sending it to the more appropriate owner) only exists because the data product's metadata names the business owner of customer consent, who can fix the problem at the source in their own small and deliberate loop.
The screenshots show what happened in our scenario. Until July, a big orange button accepted everything while managing your preferences was a tiny link, and about 80% of visitors accepted. Since August, rejecting is as easy as accepting, and only 48% of visitors still accept. The visitors and the orders stayed the same, but far fewer visits were tracked, which is why the rate seemed to double.
Merely adding a "human in the loop" can give us a false sense of comfort, and it ignores decades of psychology that have taught us how irrational we can be. It is therefore crucial that we take these old lessons into account when we design our AI systems.
So put structure first, because contracts and shared definitions make intuition trustworthy. Build deliberate loops, so that expert intuition works within a scope it knows. And wake up System 2 when certain thresholds are met, with one specific question rather than a wall of fluent text.
[1] J. Tien et al., ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents (2026), arXiv
[2] N. Gardels, Former Google CEO Eric Schmidt Maps AI’s Rapid Advance (2024), Noema Magazine
[3] C. Capuano, Can't pay attention? You're not alone (2023), University of California
[4] G. Lowery, Pattern in movies mimics that found in our brain (2010), Cornell Chronicle
[5] Princeton University, Pre-read author Maryanne Wolf urges incoming class to read deeply, with courage and effort (2026), Princeton University
[6] Faros Research, The AI Productivity Paradox Report 2025 (2025), Faros AI
[7] D. Kahneman, Thinking, Fast and Slow (2011), Farrar, Straus and Giroux
[8] D. Gilbert, R. Tafarodi and P. Malone, You can't not believe everything you read (1993), Journal of Personality and Social Psychology
[9] Bitol, Open Data Product Standard (2026), Linux Foundation
[10] Bitol, Open Data Contract Standard (2026), Linux Foundation
[11] Apache Software Foundation, Ossie, formerly Open Semantic Interchange (2026), Apache Incubator
Its expressiveness will give you goosebumps.
January 24, 20225 min read
Impact of Large Language Models on E-Commerce
April 29, 20247 min read
The simple shift in training that unlocks foresight, faster inference, and better reasoning.
November 28, 202511 min read
Reduce Noise in Sensors to better predict Power Production of Solar Panels
September 15, 20205 min read
Binary and multi-class classification, decision boundaries, discriminant functions and kNN
December 23, 20207 min read
These should not be overlooked.
You need an efficient method to adopt the rich selection of tools.
June 29, 20214 min read
My first exposure to the Logistic Regression algorithm The churn rate, also known as the...
November 3, 20206 min read
I learned how to do a regression analysis in R using brute force. With these 7 copy and paste steps, you can too.
July 27, 202013 min read
原文为英文,正文由自动抓取并保留原文。
原文发布于 Towards Data Science(Why Human in the Loop AI Oversight Fails and How to Fix It),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。