改动一条蒙提霍尔规则,五个 LLM 仍选换门

蒙提霍尔问题的各种变体会骗过 Qwen2.5 等小型 LLM
这个问题你早就知道答案了。
你正在参加一个游戏节目。现场有三扇门,其中一扇门后是一辆汽车,另外两扇门后是山羊。你选了 1 号门。知道汽车在哪里的主持人打开了 3 号门,给你看了一只山羊。然后他问:你想换成 2 号门吗?
换。换门在三次中能赢两次。1990 年 Marilyn vos Savant 在 Parade 杂志上刊出这个答案时,数千名读者写信告诉她她错了,其中不少人还是数学家 [1][2]。她并没有错。
如今没人再争论这件事,AI 模型也不争。问一个模型,它会说「换」。
于是我把其中一条规则改了。如果主持人并不知道汽车在哪里,只是碰巧打开了有山羊的那扇门呢?如果他只在你已经选中汽车时才提出换门呢?同样的门,同样的山羊,同样的问题。在第二个版本里,换门每一次都会输。
就这个谜题的六个版本,我问了五个 AI 模型该怎么做。它们几乎对每一个版本都回答「换」。包括那个换门不可能赢的版本。
让我在意的不是它们答错了,而是它们听起来有多笃定,以及我读它们自己的推理过程时看到的样子。有一个案例里,模型准确无误地把主持人的规则复述给我,然后把它忽略了。
下面是我测试它的方法,以及一个能解释这个谜题所有版本的数学原理。
关键在于主持人为什么打开了那扇门
通常的解释是这样的。你第一次选对的概率是 1/3。主持人打开一扇门并不会改变这一点。所以另一扇门正确的概率必定是 2/3。这个论证很漂亮,也确实成立,但它成立得有点偶然。曾就此撰文的统计学家 Jeffrey Rosenthal 称其为“不牢靠”的解法,因为只要谜题稍有变化,它就会失效 [1]。
真正站得住脚的,是另一个问题:鉴于主持人刚刚的举动,如果汽车在 1 号门后面,他有多大概率会这么做?在 2 号门后面呢?在 3 号门后面呢?Rosenthal 把这称为比例性原则(Proportionality Principle)[1]。在你看到某件事发生之后,每种可能性的新概率,与该事件在该可能性下发生的概率成正比。这就是去掉冗杂表述的贝叶斯法则。
P(汽车在 c 号门后 ∣ 主持人的举动) ∝ P(主持人的举动 ∣ 汽车在 c 号门后)
我们来看经典版本。你选择 1 号门,主持人打开了 3 号门。如果汽车在 1 号门后面,主持人有两扇山羊门可选,所以他有一半的概率会打开 3 号门。如果汽车在 2 号门后面,他别无选择,所以每次都会打开 3 号门。如果汽车在 3 号门后面,他根本不能打开它,所以概率为零。这样得到的权重是 1/2、1 和 0。把它们按比例缩放,使其总和为 1,就得到 1/3、2/3 和 0。这就是那个著名的答案,而它完全来自主持人的行为方式,与门本身无关。
也就是说,只要改变主持人的行为,答案就会随之改变。同样的两步,但概率那一栏不同:
如果汽车在 1、2、3 号门后面,主持人这样做的概率
知道汽车在哪里,打开一扇山羊门(如果两扇都是山羊则随机开)。我们看到的是 3 号门。
不知道。随机打开一扇门,恰好是山羊。我们看到的是 3 号门。
打开一扇门,且仅当你选中的是汽车时才提供交换。我们看到的是 3 号门以及这个交换提议。
打开编号最小的山羊门。我们看到的是 3 号门。
规则相同。我们看到的是 2 号门。
什么门都不打开,只问「想换到 2 号门吗?」
Fall 和 Crawl 出自 Rosenthal [1]。「Monty from Hell」是人们用来称呼那种只在你已经选中汽车时才提供交换的主持人。沉默的主持人是我自己加的:一个什么门都不打开、只问你要不要交换的主持人。看最后两列。换门从总是赢变成从不赢,而没有任何一扇门变动。改变的只是主持人的规则。
整个计算过程很小,可以直接贴在这里。我把它当作下文所有内容的基准真值,并且还通过每种变体模拟 300,000 局游戏来逐一核对每个答案。两者结果一致。
from fractions import Fraction as F from itertools import combinations def knowing_host(N, k): """主持人知道车在哪里,并打开 k 扇山羊门(绝不打开 1 号门)。""" def policy(c): # c = 藏车的门 goats = [d for d in range(2, N + 1) if d != c] combos = list(combinations(goats, k)) return {frozenset(s): F(1, len(combos)) for s in combos} return policy def posterior(policy, N, seen): """P(车在每扇门后 | 我们看到主持人做了 `seen`)。""" w = {c: F(1, N) * policy(c).get(seen, F(0)) for c in range(1, N + 1)} total = sum(w.values()) return {c: w[c] / total for c in w} print(posterior(knowing_host(3, 1), 3, frozenset({3}))) # {1: Fraction(1, 3), 2: Fraction(2, 3), 3: Fraction(0, 1)}
我使用了 Qwen2.5 的指令微调版本,涵盖四个规模(0.5B、1.5B、3B 和 7B,其中 7B 以 4-bit 加载)[3],另外还加上了 Phi-3.5-mini——一个来自不同家族的 3.8B 模型,以免结论只是 Qwen 特有的怪癖 [4]。对于这六个故事中的每一个,我都描述了游戏,用直白的语言说明主持人的规则,然后给出三个陈述供选择:换门更好、坚持更好,或者没有区别。
我没有让模型写出答案,而是直接看它们给答案选项字母(A、B 或 C)分配的概率。模型往往有自己偏爱的字母,所以我把每个问题以全部六种可能的排列顺序都问了一遍,再把结果按每个陈述的含义平均回去。这样一来,“总是选 A”的影响就被抵消了。
有一个数字值得记住:在这六个故事上,一个每次都回答“换”的模型会答对两个、答错四个,也就是 33%。
每个模型对每个变体回答“换更好”的概率。橙色列是换并非正确答案的变体。图片由作者提供
在经典谜题上,回答“换更好”的概率在 0.5B 上是 59%,1.5B 上是 81%,其他三个基本上都是 100%。行吧,这是正确答案。这也正是一个把它背下来的模型会给出的表现。
其他变体才让人不安。以“换”是错误答案的那四个变体为例(Monty Fall、Monty from Hell、Crawl opening door 2,以及沉默的主持人)。模型给正确陈述的平均概率是:0.5B 为 13%,1.5B 为 4.5%,3B 为 0.0%,Phi 为 0.4%,7B 为 3.7%。随机猜测都能拿到 33%。所以更大并不意味着更能察觉。更大意味着更确信什么都没变。
我的第一反应是不相信自己的代码
一片接近 100% 的结果在我看来不对劲。模型通常不会对任何事情如此一致,所以我花了一些时间试图推翻自己的结果。
是不是实验设置本身就有问题?我用同样的 A/B/C 格式加了六个简单问题(哪个数字更大、钥匙最后落在哪里之类)。7B 和 Phi 全都答对了,3B 答对 91%,1.5B 答对 85%。0.5B 只做到 52%,勉强比瞎猜强一点,所以它的数字我不太当真。我还用完整的 32 位精度重跑了 3B,以防 16 位舍入在捣鬼。答案一样。
然后是措辞。当我把问题从“换更好吗?”改成“哪扇门更可能藏着车?”时,1.5B 在经典谜题上回答“换”的比例从 81% 掉到了 38%。所以小模型的部分行为只是在对着词做反应。3B 不受影响,仍保持在约 100%。
我还想知道,它们能否运用别人告知它们的事实。在一项对照测试中,一位诚实的主持人悄悄告诉你汽车在1号门后面,也就是你自己选的那扇门,然后提出交换。答案显然是保持不动。3B 说还是换吧,比例约为100%。Phi 有67%的时候这么说。7B 只有3.5%给出了“换”,其余分别给了“保持”(48%)和“没区别”(49%)。所以,当事实被明确告知时,7B 是能运用它的,这让它在其他情况下的失误更耐人寻味。
再回到改写后的问题。用“哪扇门更有可能”这种措辞时,7B 在 Monty Fall 场景下有74%给出“同样可能”,在沉默主持人场景下是66%。两者都答对了。但在 Monty from Hell 场景下,它仍然有99.8%选择了被提供的那扇门,在 Crawl 打开2号门的场景下则是100%。这些恰恰是主持人的规则本身就构成证据的情形。“主持人没告诉我任何有用信息”,它还勉强应付得来。“主持人的行为就是线索”,它做不到。
我原以为问题出在游戏节目的设定上
我事先的猜测是,完整的 Monty Hall 设定——门、山羊和主持人——才是触发它背下答案的原因。于是我把对照实验写成了朴素的版本:三个盒子里有一个奖品,一位朋友要么如实告诉你奖品在哪,要么完全不知情。事实相同,没有游戏节目。
在三组配对场景下给出正确答案的概率,含与不含 Monty Hall 设定。图片由作者提供
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Change One Monty Hall Rule and Five LLMs Still Say Switch)
Monty Hall Problem Variants Fool Small LLMs Like Qwen2.5
You already know the answer to this one.
You're on a game show. There are three doors. Behind one is a car, behind the other two are goats. You pick door 1. The host, who knows where the car is, opens door 3 and shows you a goat. Then he asks: do you want to switch to door 2?
Yes. Switching wins two times out of three. When Marilyn vos Savant printed that answer in Parade in 1990, thousands of readers wrote in to tell her she was wrong, and plenty of them were mathematicians [1][2]. She wasn't.
These days nobody argues, and the AI models don't either. Ask one and it says "switch."
So I changed one rule. What if the host doesn't know where the car is, and just happens to open a goat door? What if he only ever offers a swap when you've already picked the car? Same doors, same goats, same question. In that second version, switching loses every single time.
I asked five AI models what to do across six versions of the puzzle. They said "switch" to almost all of them. Including the one where switching can't win.
What bothered me wasn't that they were wrong. It was how sure they sounded, and what their own reasoning looked like when I read it. In one case a model quoted the host's rule back to me, correctly, and then ignored it.
Here's how I tested it, and the one piece of math that explains every version of the puzzle.
It's about why the host opened that door
Here's the usual explanation. Your first pick is right 1/3 of the time. The host opening a door doesn't change that. So the other door has to be right 2/3 of the time. It's a nice argument and it works, but it works a bit by accident. Jeffrey Rosenthal, a statistician who has written about this, calls it the "shaky" solution, because it falls apart on slight variations of the puzzle [1].
What holds up is a different question: given what the host just did, how likely was he to do that if the car were behind door 1? Behind door 2? Behind door 3? Rosenthal calls this the Proportionality Principle [1]. After you see something happen, the new probability of each possibility is proportional to how likely that thing was to happen under it. It's Bayes' rule without the clutter.
P(car behind c∣what the host did)∝P(what the host did∣car behind c)P(\text{car behind } c \mid \text{what the host did}) \;\propto\; P(\text{what the host did} \mid \text{car behind } c)P(car behind c∣what the host did)∝P(what the host did∣car behind c)
Let's do the classic version. You pick door 1 and the host opens door 3. If the car is behind door 1, the host had two goat doors to choose from, so he'd open door 3 half the time. If it's behind door 2, he had no choice, so he'd open door 3 every time. If it's behind door 3, he can't open it at all, so zero. That gives weights of 1/2, 1 and 0. Scale them so they add up to 1 and you get 1/3, 2/3 and 0. That's the famous answer, and it came entirely from how the host behaves, not from anything about the doors.
Which means you can change the answer by changing the host. Same two steps, different column of probabilities:
Chance the host does this if the car is behind door 1, 2, 3
Knows where the car is, opens a goat door (random if he has two). We see door 3.
Does not know. Opens a random door, which happens to be a goat. We see door 3.
Opens a door and offers a swap only if you picked the car. We see door 3 and the offer.
Opens the lowest-numbered goat door. We see door 3.
Same rule. We see door 2.
Opens nothing, just asks "want to swap to door 2?"
Fall and Crawl are Rosenthal's [1]. "Monty from Hell" is what people call a host who only offers the swap when you've already picked the car. The silent host is my own addition: a host who opens nothing and just asks if you want to swap. Look at the last two columns. Switching goes from always winning to never winning, and not a single door moved. Only the host's rule changed.
The whole calculation is small enough to paste here. I used it as the ground truth for everything below, and I also checked every answer by simulating 300,000 games per variant. They match.
from fractions import Fraction as Ffrom itertools import combinationsdef knowing_host(N, k): """Host knows where the car is and opens k goat doors (never door 1).""" def policy(c): # c = door hiding the car goats = [d for d in range(2, N + 1) if d != c] combos = list(combinations(goats, k)) return {frozenset(s): F(1, len(combos)) for s in combos} return policydef posterior(policy, N, seen): """P(car behind each door | we saw the host do `seen`).""" w = {c: F(1, N) * policy(c).get(seen, F(0)) for c in range(1, N + 1)} total = sum(w.values()) return {c: w[c] / total for c in w}print(posterior(knowing_host(3, 1), 3, frozenset({3})))# {1: Fraction(1, 3), 2: Fraction(2, 3), 3: Fraction(0, 1)}
I used the instruction-tuned versions of Qwen2.5 at four sizes (0.5B, 1.5B, 3B and 7B, with the 7B loaded in 4-bit) [3], plus Phi-3.5-mini, a 3.8B model from a different family, so this wouldn't turn out to be a Qwen quirk [4]. For each of the six stories I described the game, with the host's rule spelled out in plain words, and then gave three statements to choose from: switching is better, staying is better, or it makes no difference.
I didn't let the models write an answer. I looked at the probabilities they assigned to the answer letters (A, B or C) directly. Models tend to have favorite letters, so I asked every question in all six possible orderings and averaged the results back onto what each statement meant. That way "always picks A" washes out.
One number to keep in your head: on these six stories, a model that says "switch" every single time gets two right and four wrong. That's 33%.

Probability each model answers "switching is better" for each variant. Orange columns are the variants where switching is not the right answer. Image by author
On the classic puzzle, the probability on "switching is better" was 59% for the 0.5B, 81% for the 1.5B, and basically 100% for the other three. Fine, that's the right answer. It's also exactly what you'd see from a model that memorized it.
The other variants are where it gets uncomfortable. Take the four where "switch" is the wrong answer (Monty Fall, Monty from Hell, Crawl opening door 2, and the silent host). The average probability the models gave to the correct statement was 13% for the 0.5B, 4.5% for the 1.5B, 0.0% for the 3B, 0.4% for Phi and 3.7% for the 7B. Random guessing would get 33%. So bigger didn't mean better at noticing. Bigger meant more sure that nothing had changed.
My first reaction was to distrust my own code
A wall of near-100% looked wrong to me. Models are not usually this consistent about anything, so I spent some time trying to break my own result.
Was the setup just broken? I added six trivial questions in the same A/B/C format (which number is bigger, where did the keys end up, that kind of thing). The 7B and Phi got all of them, the 3B got 91% and the 1.5B got 85%. The 0.5B only managed 52%, which is barely better than guessing, so I wouldn't read much into its numbers. I also reran the 3B in full 32-bit precision, in case 16-bit rounding was doing something strange. Same answers.
Then wording. When I asked "which door is more likely to hide the car?" instead of "is switching better?", the 1.5B's "switch" on the classic puzzle dropped from 81% to 38%. So some of what the small models do is just reacting to words. The 3B didn't care and stayed at about 100%.
I also wanted to know whether they can use a fact that's handed to them. In one control, an honest host quietly tells you the car is behind door 1, your own door, and then offers a swap. The answer is obviously to stay. The 3B said switch anyway, at about 100%. Phi said it 67% of the time. The 7B only gave "switch" 3.5%, splitting the rest between "stay" (48%) and "no difference" (49%). So the 7B can use a fact when it's told one, which makes its other failures more interesting.
Back to the reworded questions for a second. With the "which door is more likely" wording, the 7B gave 74% to "equally likely" for Monty Fall and 66% for the silent host. Both correct. But it still picked the door it was offered at 99.8% for Monty from Hell and 100% for Crawl opening door 2. Those are the cases where the host's rule is the evidence. "The host told me nothing useful" it can sort of handle. "The host's behavior is the clue" it can't.
I thought the game-show setup was the problem
My guess going in was that the full Monty Hall setup, with doors and goats and a host, was what triggered the memorized answer. So I wrote plain twins of the controls: a prize in one of three boxes, and a friend who either tells you the truth about where it is or has no idea. Same facts, no game show.
Probability of the correct answer on three matched pairs, with and without the Monty Hall setup. Image by author
It barely mattered. The 3B scored 33% on both versions. Phi went from 41% to 46%. The 7B went from 49% to 60%, which is something, but not a lot. Even with the friend saying the prize is in box 1, the 3B said "swap to box 2" 99.9% of the time.
So it isn't the doors and goats. It's the shape of the situation: three options, you pick one, someone offers you a swap. My plain version has that shape too, so I can't say where the pull stops. I can only say that taking off the costume wasn't enough.
Letting them think out loud
Fair objection: nobody solves a probability puzzle in one instant. So I let the chat models write out their reasoning first (five sampled chains per story, up to 600 tokens each, 55 chains per model) and read the final answer off the end.
It didn't help. Across the 30 chains on the six main variants, the models ended on "switch" 80% of the time for the 0.5B, 97% for the 1.5B, 100% for the 3B, 100% for Phi and 97% for the 7B. Accuracy came out at 33%, 30%, 33%, 33% and 37%. Which is, give or take, what always saying "switch" gets you.
I read a lot of these chains, and they're the best part of the whole experiment. In the Monty from Hell story the host only offers a swap if you picked the car. One 7B chain states that rule correctly (if you had picked a goat, "the host will not offer a switch") and then a few lines later decides that switching is better "if you initially picked a goat, which is more likely." It had the prior and it had the evidence, and it went with the prior. Nowhere does it ask how likely the host was to make this offer in each case, which is the whole point of the table above.
In the Monty Fall story the host explicitly does not know where the car is. One 3B chain starts with "Since the host knows where the car is," which is the classic puzzle, not the one it was given. It had quietly rewritten the problem into the one it remembered. And in the plain box version, a 7B chain says box 2 "still has a 0 chance if the prize is indeed in box 1 as indicated by your friend," and then answers "switch", to box 2.
One more thing I can't fully explain. For the box story where the friend says the prize is in box 1, the 7B's instant answer leaned "stay" at 69%. After writing out its reasoning, all five chains said "switch." The prompts are worded a bit differently and the chains are sampled, so I wouldn't call it proof that thinking made it worse. It's still the opposite of what I was hoping to see.
One last test, because the math here is pretty. With 10 doors and a host who knowingly opens k goat doors, switching to a particular remaining door wins with probability (N−1)/(N(N−1−k))(N-1) / (N(N-1-k))(N−1)/(N(N−1−k)), while staying wins 1/N1/N1/N.
P(win by switching)=N−1N(N−1−k),P(win by staying)=1NP(\text{win by switching}) = \frac{N-1}{N\,(N-1-k)}, \qquad P(\text{win by staying}) = \frac{1}{N}P(win by switching)=N(N−1−k)N−1,P(win by staying)=N1
At k = 0 the host opened nothing and the two options tie. By k = 8, switching is nine times better.
Model confidence in "switch" as the host opens more doors, against the true advantage of switching. Image by author
The 7B got k = 0 right: 51% switch, basically a shrug. Then at k = 1 it jumped to 99% and stayed between 84% and 96% for every bigger k, never climbing as switching got more and more obviously right. The 3B and Phi said "switch" at 99.9% or more even when the host opened nothing. A model following the math would get more confident with every door. These behaved like a light switch, not a dial.
What I would and wouldn't claim
It's a small study. The models run from 0.5B to 7B, one of them is quantized, four are from the same family, and I only looked at quick answers and short sampled chains. Six stories, a few controls, five chains per story. My counts of chains that rewrite the puzzle come from reading text and a simple pattern search, so treat those as examples, not rates. I didn't test any frontier models.
There is a benchmark that did look at bigger ones. It took 25 classic puzzles, made 149 altered versions, and tested eleven leading models, closed ones included. It reports the same kind of failure: confidently giving the memorized solution to a puzzle that no longer matches it [5]. That lines up with what I saw, but I haven't reproduced it on big models, so I'm not going to claim it.
What I will say is narrower. For these models, getting a famous puzzle right is weak evidence of reasoning, because the same answer shows up whether or not the puzzle changed. The cheapest test I know is to change one rule so the right answer flips, and see whether the answer flips with it.
Next time a model gets a famous puzzle right, I'll be changing one rule before I get impressed.
[1] Jeffrey S. Rosenthal, Monty Hall, Monty Fall, Monty Crawl (2008), Math Horizons, September 2008, pp. 5-7
[2] Marilyn vos Savant, Ask Marilyn (1990), Parade Magazine, September 9, 1990, p. 22
[3] Qwen Team (An Yang et al.), Qwen2.5 Technical Report (2024), arXiv:2412.15115
[4] Marah Abdin et al., Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024), arXiv:2404.14219
[5] Souradeep Mukhopadhyay, Rishabh Baral, Nimeesh Mahajan, Samhitha Harish, Aswin RRV, Mihir Parmar, Mutsumi Nakamura, Chitta Baral, Phantom Recall: When Familiar Puzzles Fool Smart Models (2025), arXiv:2510.11812
A comparison of frameworks, models, and costs
October 30, 202415 min read
An applied project detailing the performance of LLMs for geocoding, in comparison to modern geocoding APIs.
September 13, 20239 min read
Explore various prompt engineering techniques for arithmetic reasoning problems, best practices, and rapid experimentations for...
November 18, 202315 min read
A meta analysis that turns out positive yet identifies the need for further research
Our weekly selection of must-read Editors' Picks and original features
November 16, 20233 min read
Visualisation of the pre-processing steps natively in Scikit-Learn 0.23 version(May 2020) is a productivity booster.
August 31, 20204 min read
Being good at coding is simply not enough
November 21, 20219 min read
Evaluating RAGs on graph and SQL databases
November 1, 20257 min read
Introduction to key concepts you need to understand for an ARIMA time series project.
August 27, 20228 min read
原文为英文,正文由自动抓取并保留原文。
原文发布于 Towards Data Science(Change One Monty Hall Rule and Five LLMs Still Say Switch),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。