干扰段落增至268万,RAG 检索掉了40分

BM25 RAG 检索测试:文档规模从 2,000 份到 268 万份
在一个包含 100 篇研究论文的集合上,标准文本搜索方法 BM25 在 99.7% 的问题中,把能回答该问题的论文放进了前五条结果。在 1,801 篇论文上,这一比例为 98%。我设计这项测试,是为了找到该比例开始下降的阈值,而它几乎没有变化。
对任何运行检索增强生成(RAG)的人来说,这个结果看上去是好消息——在 RAG 中,AI 模型利用搜索步骤先行取回的段落来回答问题。如果随着集合规模增长搜索依然表现良好,系统其余部分就有了坚实的立足点。随后我在 268 万条 Wikipedia 段落上跑了同类测试,情况就变了。
本文把这两项测试作为一个完整故事来呈现。第一项测试使用的是我此前智能体式检索测试中的问题。第二项使用的是公开问题,这些问题并非为我的测试环境而编写。
每项测试的代码、问题、标注和结果都放在一个公开仓库中。
我想弄清楚什么?
我想知道,随着集合规模增长,RAG 系统的搜索步骤——也就是负责找到文档的那一环——有多频繁地漏掉正确文档。我还想知道,如果多出来的文档看起来像正确答案,这会不会有影响。
干扰文档是指不能回答该问题的文档。有些干扰文档是随机的,有些则在主题上与正确文档很接近。
每个测试都问同一个直白的问题。如果我增加更多干扰文档,其他什么都不改,检索是否仍能找到回答该问题的文档?
在运行任何测试之前,我先定下一条规则,规定什么才算有意义的下降,这样我就不会在看到结果后再挑选标准。某个规模只有在其分数比最小集合低至少五分时,才被算作一个退化点。五分意味着每 20 个问题中多一个失去答案,而这里的 1 分指 1 个百分点。
这条规则还要求下降必须确凿真实,而非运气。我的测试只使用部分问题,所以换一组问题会得到略有不同的下降。我用 95% 区间来检查这一点,这个区间是真实下降可能所在的跨度范围。
想象一台可能有约 2 公斤误差的体重秤,上下浮动都可能。如果它说你减了 1 公斤,你无法判断自己到底有没有减重。如果它说你减了 8 公斤,那你确实减了。
我测得的下降就是秤上的读数,而 95% 区间就是这台秤的波动幅度。如果这个跨度包含零(即没有变化),那么下降就可能是运气。
例如,6 分的下降,如果其跨度从下降 9 分到上升 1 分,就不算数,因为这个跨度包含没有变化。6 分的下降,如果其跨度从下降 9 分到下降 3 分,就算数,因为这个跨度中的每个值都是下降。
本文出现三个分数:recall@5、success@10 和 exact match。前两个问的是检索是否把好的段落放在靠前位置,exact match 问的是模型的答案是否与某个可接受的答案匹配。我会在它们首次起作用的地方分别定义。
测试一,2,000 篇 arXiv 论文
第一个集合收录了 2,000 篇近期 arXiv 上关于机器学习、语言和搜索的摘要。我使用了 298 个问题,每个问题都依据一篇论文撰写并逐一手工核对,因此每个问题都有已知的正确答案论文。没有论文被使用两次,所以这 298 个问题来自 298 篇不同的论文。
针对每个问题,我都构建了包含 100、250、500、1,000 和 1,801 篇论文的集合,且每个更大的集合都包含较小的那个。正确答案论文始终在其中。其余的是两类干扰项。
这 2,000 篇中的其他任何论文都可以用作干扰项,包括 1,702 篇没有对应问题的论文,以及属于其他问题的 297 篇论文。对任何一个问题而言,这些论文都是不应回答该问题的文档,我接下来对此进行了核查。
普通干扰项是从与正确答案论文不接近的论文中随机抽取的。我做了三次彼此独立的随机抽取,我称之为 draw。
近似干扰项是与正确答案论文最接近的论文,按相似度顺序依次选取。我用一个单独的小型嵌入模型来衡量相似度,这种工具能把文本转换成数字列表。这样一来,这些集合就不是为了糊弄我正在测试的检索方法而构建的。
我尝试了两种检索方法:BM25 和嵌入向量。嵌入向量按照段落的数字列表与问题的数字列表的接近程度来对段落排序。
我用两种方式运行每种方法,分别称为扁平块(flat chunks)和父文档(parent documents),这样就构成了四种检索设置。扁平块检索的是两个句子的小窗口,并返回排名前五的窗口。父文档检索同样的窗口,但返回排名前五的不同论文的完整摘要。
这四种设置是:BM25 配扁平块、BM25 配父文档、嵌入向量配扁平块,以及嵌入向量配父文档。
这项评分是 recall at 5(前 5 召回率),即在前五条结果中出现了能回答该问题的论文的问题所占的比例。一篇论文只有通过我接下来描述的检查,才会被计为答案。
我如何检查干扰项没有回答该问题
相似论文可能会碰巧回答该问题,这会让搜索表现看起来比实际更差。因此我用两个 AI 模型对每个问题最接近的 50 篇论文进行筛查。
Claude Haiku 5.5 标记出可能回答该问题的论文,Claude Sonnet 5.5 则复核每一个标记。Haiku 标记了 446 个案例,Sonnet 判定其中 313 个没有问题,但为保险起见,我把所有被标记的论文都移除了。
每篇被移除的论文都由次近的一篇替补,因此各集合的规模保持精确。移除用掉了 2,000 篇论文中的一部分,所以最大的集合是 1,801 篇而不是 2,000 篇。有一个问题被两个 AI 模型中的一个拒绝处理,于是我手动检查了它最接近的论文。
之后我亲自阅读了部分标注样本以核对模型的标注,其中包括模型从未看过的 40 篇论文,这 40 篇全部明显不相关。我并没有阅读每一对,所以我把这些标注视为一个大致的范围。调用这两个 AI 模型总共花费了 1.81 美元。
表 1. 筛查之后在 298 个问题上的 recall at 5(百分比)。变化量为最小集合与最大集合之间的差值,负数表示评分下降。相似干扰项只运行了一次,因为它们没有随机抽样;普通干扰项则取三次运行的平均值。
图 1. 在 100 篇论文和 1,801 篇论文时的 recall at 5。横轴从 92% 起,因此差距看起来比实际更大。
没有任何一种设置达到我的规则标准。最大的降幅是 3.4 分,出现在使用扁平分块和普通干扰项的 embeddings 上。这低于我设定的 5 分门槛。
但 3.4 只是我在自己那 298 个问题上测到的结果。假设你想知道一座城市里成年人的身高,于是测量了 298 个人,他们的平均身高是 170 厘米。
你不会说整座城市的平均身高恰好是 170,因为换另外 298 个人来测,得到的数字会略有不同。你会说它大概在 168 到 172 之间。
我测到的降幅也是同样的道理。3.4 就是我的 170,而 95% 区间就是我的 168 到 172。就这一设置而言,区间从 1.3 分延伸到 5.4 分。
该区间的上端高于 5 分,所以我无法排除真实降幅刚好越过我门槛的可能。我只能说,我测到的降幅从未达到这个门槛。
在 1,801 篇论文的规模下,父文档在全部四组对比中都只比扁平分块高 0.3 分,这四组指的是 BM25 和 embeddings 各自搭配每一种干扰项。这些区间大幅重叠,所以我不会把这称为一次胜出。它们也没有明显减轻规模增长带来的损害,因为降幅在两组对比中更小,在一组中更大,在一组中相同。
当我只移除两个模型都标记出的那 133 篇论文时,结果依然成立;当我剔除三个其论文实际上并不支持答案的问题时,结果也依然成立。当我剔除那八个只需从一篇论文中获取一个事实的问题时,结果同样成立,因为没有任何论文能部分回答它们。在每一个版本中,最大降幅都保持在 3.1 到 3.4 分之间。
综合来看,随着语料库规模增长,我测试的这些检索设计选择表现并不一致。这些选择包括检索方法——BM25 还是 embeddings,以及检索返回的内容——扁平分块还是父文档。
在面对普通干扰项时,嵌入的得分损失比 BM25 更大,但在面对相似干扰项时并不明显。父文档也没有明显胜过扁平分块。这只是对两种检索方法的一次测试,所以应当把它当作测试你自己配置的理由,而不是一条规则。
有一个细节很重要。相似干扰项集合的起始得分更低,在最小规模时为 96.0 到 98.3,这大概是因为即便只有 99 篇相似论文,也已经会和正确的那一篇激烈竞争。扩大集合规模只多损失了一两分。
为什么那项测试感觉太容易了?
那项测试几乎没有留出多少下降空间。得分一开始就接近 100,集合规模最多到 2,000 篇论文,而我能为两类干扰项构建的最大集合是 1,801 篇。而真实的文档库可能要大得多。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(RAG Retrieval Lost 40 Points When Distracting Passages Grew to 2.68M)
BM25 RAG Retrieval Tested From 2,000 to 2.68M Documents
On a collection of 100 research papers, BM25, a standard text search method, put a paper that answered the question in its top five results for 99.7 percent of questions. On 1,801 papers, that number was 98 percent. I built the test to find the threshold at which that number drops, and it barely moved.
That result looked like good news for anyone running retrieval augmented generation, or RAG, where an AI model answers a question using passages that a search step fetched first. If search stays good as the collection grows, the rest of the system has solid ground to stand on. Then I ran the same kind of test on 2.68 million Wikipedia passages, and the picture changed.
This article reports both tests as one story. The first test used questions from my earlier test of agentic retrieval. The second used public questions that nobody wrote for my setup.
The code, questions, labels, and results for every test are in a public repository.
What did I want to find out?
I wanted to know how often the search step of a RAG system, the part that finds documents, misses the right one as the collection grows. I also wanted to know whether it matters if the extra documents look like the right answer.
A distractor is a document that does not answer the question. Some distractors are random, and some are close in topic to the right document.
Each test asked the same plain question. If I add more distractors and change nothing else, does the search still find a document that answers the question?
Before running anything, I fixed a rule for what would count as a meaningful drop, so that I could not choose the bar after seeing the results. A size counted as a degradation point only if the score fell at least five points below the smallest collection. Five points means one more question in every 20 loses its answer, and a point here means one percentage point.
The rule also needed the drop to be clearly real and not luck. My tests use only some questions, so a different set of questions would give a slightly different drop. I checked this with the 95 percent range, which is the span where the true drop probably sits.
Think of a bathroom scale that can be wrong by about 2 kilograms either way. If it says you lost 1 kilogram, you cannot tell whether you lost anything. If it says you lost 8 kilograms, you really did.
My measured drop is the scale reading, and the 95 percent range is how jumpy the scale is. If that span includes zero, which means no change, the drop could be luck.
For example, a 6 point drop with a span from a 9 point drop to a 1 point gain does not count, because that span includes no change. A 6 point drop with a span from a 9 point drop to a 3 point drop counts, because every value in that span is a drop.
Three scores appear in this article: recall at 5, success at 10 and exact match. The first two ask whether search put a good passage near the top, and exact match asks whether the model's answer matched an accepted answer. I define each one where it first matters.
Test one, 2,000 arXiv papers
The first collection held 2,000 recent arXiv abstracts in machine learning, language, and search. I used 298 questions, each written from one paper and checked by hand, so every question had a known right paper. No paper was used twice, so the 298 questions came from 298 different papers.
For each question I built collections of 100, 250, 500, 1,000 and 1,801 papers, and each larger collection contained the smaller one. The right paper was always inside. The rest were distractors of two kinds.
Every other paper in the 2,000 could be used as a distractor. That included the 1,702 papers with no question and the 297 papers that belonged to other questions. For any one question, those papers were documents that should not answer it, and I checked that next.
Ordinary distractors were picked at random from papers that were not close to the right one. I made three separate random picks, which I call draws.
Look-alike distractors were the papers nearest to the right one, taken in order of similarity. I measured similarity with a separate small embedding model, a tool that turns text into lists of numbers. That way the collections were not built to fool the search methods I was testing.
I tried two search methods, BM25 and embeddings. Embeddings rank passages by how close their lists of numbers are to the question's list.
I ran each method in two ways, called flat chunks and parent documents, which makes four search setups. Flat chunks searched small windows of two sentences and returned the top five windows. Parent documents searched the same windows but returned the whole abstract for the top five distinct papers.
The four setups are BM25 with flat chunks, BM25 with parent documents, embeddings with flat chunks and embeddings with parent documents.
The score was recall at 5, which is the share of questions where a paper that answers the question appears in the top five results. A paper counted as an answer only if it passed a check I describe next.
How I checked that distractors did not answer the question
A look-alike paper can accidentally answer the question, which would make the search look worse than it is. So I used two AI models to screen the 50 closest papers for each question.
Claude Haiku 5.5 flagged papers that might answer the question, and Claude Sonnet 5.5 reviewed each flag. Haiku flagged 446 cases and Sonnet cleared 313 of them, but I removed every flagged paper to be safe.
Each removed paper was replaced by the next closest one, so collection sizes stayed exact. Removals used up part of the 2,000 papers, which is why the largest collection is 1,801 papers and not 2,000. One question was refused by one of the two AI models, so I checked its closest papers by hand.
Afterwards I read samples of the labels myself to check the models' labels, including 40 papers the models never looked at, and all 40 were clearly off topic. I did not read every pair, so I treat these labels as a rough range. The calls to the two AI models cost 1.81 US dollars in total.

Table 1. Recall at 5 in percent on 298 questions, after screening. The change is between the smallest and the largest collection, and a negative number means the score fell. Look-alike distractors were run once, since they have no random draws, and ordinary distractors were averaged over three draws.
Figure 1. Recall at 5 at 100 papers and at 1,801 papers. The horizontal axis starts at 92 percent, so the gaps look larger than they are.
No setup met my rule. The biggest drop was 3.4 points, for embeddings with flat chunks and ordinary distractors. That is below my bar of five points.
But 3.4 is only what I measured on my 298 questions. Say you want to know how tall adults in a city are, and you measure 298 people. Their average is 170 centimeters.
You would not say the whole city averages exactly 170, because a different 298 people would give a slightly different number. You would say it is probably between 168 and 172.
My drop works the same way. The 3.4 is my 170, and the 95 percent range is my 168 to 172. For this setup the range ran from 1.3 to 5.4 points.
The top of that range is above five, so I cannot rule out a true drop just past my bar. I can only say that the drop I measured never reached it.
Parent documents ended 0.3 points higher than flat chunks in all four comparisons at 1,801 papers, meaning BM25 and embeddings with each kind of distractor. The ranges overlap heavily, so I would not call that a win. They also did not clearly reduce the damage from growth, since the drop was smaller in two comparisons, larger in one and the same in one.
The result held when I removed only the 133 papers that both models flagged, and when I dropped three questions whose paper did not actually support them. It also held when I dropped the eight questions that need only one fact from a paper, because no paper can partly answer them. In every version the largest drop stayed between 3.1 and 3.4 points.
Taken together, the retrieval design choices I tested did not behave consistently as the collection grew. Those choices were the search method, BM25 or embeddings, and what the search returns, flat chunks or parent documents.
Embeddings lost more than BM25 with ordinary distractors, but not clearly with look-alike ones. Parent documents did not clearly beat flat chunks either. This is one test of two search methods, so treat it as a reason to test your own setup and not as a rule.
One detail matters. Look-alike collections started lower, at 96.0 to 98.3 for the smallest size, probably because even 99 look-alike papers already compete hard with the right one. Growing the collection cost only one or two more points.
Why did that test feel too easy?
The test left very little room to fall. The scores began near 100, the collection stopped at 2,000 papers, and the biggest collection I could build for both kinds of distractor was 1,801. A real document store can be far larger.
The questions were also written from the paper they ask about, so they may share words with it. A search method has an easy job when the question and the answer use the same vocabulary. That could hide how it behaves on questions that people really type.
I did not want to leave the result at "no breaking point found". So I looked for public questions that nobody had written from the answer, and a much larger collection.
Test two, 2.68 million Wikipedia passages
Natural Questions contains real, anonymized search queries, with answers marked on Wikipedia pages. BEIR packages 3,452 of those questions with 2,681,468 Wikipedia passages and human relevance judgments. A relevance judgment is a label saying a passage is known to help answer a question.
A passage that carries that label is what I call a judged passage. It works like the answer key at the back of a textbook. People marked these passages before I ran any search so that I could check the search against the key.
Passages without a label are not necessarily wrong. Nobody marked them either way, so an unlabeled passage can still contain an answer.
I set aside 40 questions for checking my code and used the other 3,412 for the size test, which compares collection sizes, each with at least one judged passage. I used BM25 only. The score was success at 10, which counts a question as a hit when at least one judged passage appears in the first ten results.
I built nested collections of 10,000, 100,000, 500,000 and 1,000,000 passages, so that each larger one contains the smaller ones. Each came from three random draws, and I ran the full 2,681,468 passages once. Every collection contained all the judged passages, and the rest were filled with passages that had no label.
Table 2. Success at 10 on 3,412 Natural Questions queries with BM25. I checked every number in this table against the saved result rows.
The score fell at every size. The first size to meet my rule was 100,000 passages, with a drop of 11.1 points and a range from 10.2 to 12.1 points. At the full collection the score was about 40 points below the 10,000 passage result.
Figure 2. Share of questions with a judged passage in the first ten results, by collection size.
BM25 lost at most 2 points on the 2,000 papers and lost 40 points here. The two tests differ in many ways, including topic, question style, labels, and size, so I cannot say which difference mattered most. What I can say is that the first test did not predict the second.
Two things that make this number harder to read
First, the labels are positive only. A passage with no label might still answer the question, and a search that returns it counts as a miss. For a question about the tallest mountain, one passage may carry the label while several others also name Everest.
I saw this in the answer test described below. In 21 of the 210 ordinary question and draw pairs, no judged passage reached the first ten results. The answer model still got the right answer in 5 of them, about one case in four, so other passages must have carried it.
Second, my smallest collection is not a natural one. Every collection holds all 4,201 judged passages, which make up about 42 percent of a 10,000 passage collection and under 0.2 percent of the full one.
So part of the drop is the judged passages becoming a smaller share of the collection. That also happens in real life as a collection grows. My smallest collection starts with an unusually high share, though, and I did not run a version with a natural mix.
What happens when the extra passages look alike?
The size test changed millions of passages at once. So I ran a smaller test that changed exactly 20 passages per question. I sampled 100 questions at random from those that had at least 50 other passages from the same Wikipedia page as the judged passage.
AI screening helped sort the candidates, but I read all of the passages myself. For each question I labeled the 50 nearest same page passages as full support (answers the whole question), partial support (answers one part) or no support (answers none of it). I did this without seeing the similarity rank, and I finished before running any search, so the results could not shape my labels.
I used 50 passages per question because a pilot showed that 20 would leave too few with no support once the helpful ones were set aside.
That gave 5,006 labeled passages, of which 81 were full, 269 partial and 4,656 had no support. Only passages with no support were used as distractors. I was the only reviewer, so these labels have no second annotator.
For each question I built three pairs of 10,000 passage collections. In the ordinary collection, the filler was random, and never came from the 50 same page passages I had reviewed. In the hard collection, I swapped 20 of the filler passages for 20 passages from the answer's own page, so the two collections differed by exactly 20 passages.
Table 3. Success at 10 on 100 questions, averaged over three draws. The drop is 8.3 points, with a range from 3.7 to 13.7 points.
The drop showed in every draw, from 90 to 81 percent, 92 to 84 percent and 90 to 82 percent. Across 300 question and draw pairs, 25 changed from a hit to a miss and none changed from a miss to a hit.
Figure 3. Each pair of dots is one draw of 100 questions, ordinary filler on the right and same page passages on the left.
What I found when I read the swapped passages
As a second look at my own labels, I reread one swapped passage from each of 40 random questions. This was still my own judgment, not an independent reviewer. Of those 40, 38 clearly did not answer the question, and two were borderline.
Most of the swapped passages were short pieces of the same page, such as a line reading "League championships (5)", and they were not close in meaning. They ranked high because the search gives a title match twice the weight of a text match.
So any passage from a page such as Wake Island ranks high for a question about Wake Island.
That changes what this test shows. It shows that same page passages hurt BM25 with a doubled title weight. It does not show that passages with similar meaning would fool a search method, and I did not test embeddings here.
I also checked the 25 cases that changed from a hit to a miss. In 21, the swapped passages ranked above the judged passage and pushed it down.
In the other four, none of the 20 added passages reached the top ten, yet the judged passage still fell. BM25 ranks passages by a number that depends on how common each word is across the whole collection, so swapping 20 passages can nudge those numbers slightly. That is my guess, and I did not test it.
One question was borderline, "when did the US take over Wake Island". I labeled two swapped passages as no support, yet on a second look they describe the Navy resuming control in 1945 and the Army taking command in 1994.
That question changed in all three draws, so it may inflate the 8.3 point drop a little.
Did the answer model get worse?
Recall at 5 and success at 10 only show whether search found the right passage, not whether a model can still answer from the passages it gets. So I tested a small open model, Qwen3 4B, which I call the answer model. For the same 100 questions and three draws, I gave it the top ten results from the ordinary collection, and then the top ten from the hard collection.
I call the ten passages the answer model sees for one question and one draw a context. I used the same instructions every time and told the answer model to always pick its most likely next word, so its answers do not change between runs.
I ran the answer test on the Wikipedia questions, because the first test would have needed a paid judge for its answers. The answer model costs nothing per call.
I found public lists of accepted answers, called aliases, for 70 of the 100 questions by matching them to questions in a published Natural Questions answer file. That file comes from the Dense Passage Retrieval project. I scored those 70, and the other 30 had no accepted answers to score against.
Exact match counts an answer only when it matches an accepted answer word for word, after ignoring capital letters, punctuation and words such as "the". I averaged the three draws for each question, and the ranges come from resampling the questions many times.
Table 4. Exact match on 70 questions with answer references. In the second row, the same judged passage was placed first in both contexts, followed by nine search results that were not judged.
With the real search results, the answer model scored 11.9 points lower. When I forced the judged passage to the front, the two conditions were almost equal. Scoring the answers by word overlap, which gives partial credit, gave a drop of 12.2 points, so the choice of scorer did not drive the result.
The score depended on where the judged passage sat, not on which collection it came from. When a judged passage came first, exact match was 65.8 percent in ordinary contexts and 68.3 percent in hard ones. When it came second to fifth, the figures were 41.9 and 42.6 percent, though that covers only 31 ordinary and 68 hard contexts.
So hard search hurt mainly because it more often pushed the judged passage down or out of the top ten. Each condition has 210 contexts, from 70 questions and three draws. A judged passage came first in 152 of the 210 ordinary contexts and 82 of the 210 hard ones.
Take the question about when Teenage Mutant Ninja Turtles came out. Ordinary search put the judged passage first and the answer model said 1984, but hard search moved that passage to eighth or ninth place and the answer model said 1989.
Exact match also marked reasonable answers wrong. A complete list of host countries counted as wrong because it matched no single alias, and "1954" scored below "June 27, 1954". The result with the judged passage placed first does not prove the answer model ignores distraction, because its range allows a loss of a few points.
What should you check in your own system?
This is a guess about practice, based on one search method and one small model.
Table 5. From result to action.
If your questions are written from your documents, the search may look better than it will with real users. If you can, add questions written by people who have not seen the documents. Treat the position of the judged passage in the final context as something to measure, because the answer model did much better when the answer came first.
[1] Kwiatkowski et al., Natural Questions: a Benchmark for Question Answering Research, 2019 (CC BY-SA 3.0).
Or, When you feel your life's too hard, just go have a talk with Claude
June 9, 202416 min read
A physicist's approach to building production-grade agents
A deep dive into advanced indexing, pre-retrieval, retrieval, and post-retrieval techniques to enhance RAG performance
October 15, 202428 min read
A prompt-based experiment to improve both accuracy and transparent reasoning in content personalization.
November 30, 20247 min read
Exploring the advanced version of the attention mechanism in Transformers
November 28, 20239 min read
LEC surpasses best in class models, like GPT-4o, by combining the efficiency of a ML classifier with the language understanding of an LLM
December 20, 20248 min read
Comprehensive guide to ChatGPT API for newbies
October 4, 202317 min read
Inside disaggregated LLM inference — the architecture shift behind 2-4x cost reduction that most ML teams haven't adopted yet.
April 15, 202614 min read
A cheaper and faster unified fine-tuning technique
April 19, 20247 min read
原文为英文,正文由自动抓取并保留原文。
原文发布于 Towards Data Science(RAG Retrieval Lost 40 Points When Distracting Passages Grew to 2.68M),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。