11-19 更新 实时
首页/ 技术栈/ 正文

本地 LLM 能处理教科书式检索增强生成,却在新论文上失手

技术栈 来源:Towards Data Science 2026-11-16 20:00 0 阅读 机器翻译
本地 LLM 能处理教科书式检索增强生成,却在新论文上失手

我读大量技术材料:机器学习教材、粒子物理,以及我自己领域的论文。我想要一个能回答我书库相关问题的助手,精确标注引用页码,把原文段落展示给我,并且运行在我自己的笔记本电脑上。不上传、不订阅、不用猜答案从哪来。

老实说说是怎么做的:大部分代码并不是我自己敲的。Claude Code 在我的机器上分阶段把它搭建起来,而由我负责设计、编写测试题、阅读每一份报告,并在每个关口做出决定。这种工作方式本身就是一个故事。本文讲的是测量结果显示了什么,因为其中有些让我意外。

简短版:在教材类问题上,本地小模型表现不错。但面对今年发表的一篇论文,在检索相同、提示词相同、摆在它们面前的段落也相同的情况下,它们几乎跌到零。如果你要构建本地 RAG 系统,材料对模型而言有多新,比问题听起来有多难更重要。

询问我自己的论文时:回答会引用相应段落,查看器会在页面上将它们高亮。图片由作者提供。

问题问的是:在我那篇关于地震波场自回归序列模型的论文中,K、C、T 和 H 分别是什么意思,一个 Token 对应多长时间,以及一次 240 s 的推演需要多少个 Token。托管的 Claude Opus 给出的回答解释了每个符号(K = 每个 Token 16 个采样点,C = 3 个通道,T = 320 个上下文 Token,H = 4 个预测 Token),算出一个 Token 约为 8.4 s,整段推演约为 28 个 Token,并引用了 S1 和 S4 两段。PDF 查看器显示 Method 部分,其中被检索到的段落已高亮。在对话下方,检索范围限定为勾选的那篇论文,并列出四种回答模式及其成本。

我构建的东西,一图概览

这个应用是一个带 React 前端的 FastAPI 服务器。你把 PDF 和 PowerPoint 演示文稿放进一个文件夹。每份文档用 Docling 解析一次,独立公式由 Granite-Docling 再次读取,并与 PDF 自身的字形进行核对,文本被切成 300–600 Token 的块,绝不会拆分一个公式、一张表格或一个代码块。

一个问题是如何被回答的。一切都在一块 16 GB 的笔记本 GPU 上运行。图片由作者提供。

一个问题(追问会先由一个小型 1.7B 辅助模型重写)会进入两种搜索:在 SQLite FTS5 中用 BM25 进行的关键词搜索,以及用 bge-m3 向量进行的语义搜索。结果被合并,一个交叉编码器(cross-encoder)将前 30 个重排为 6 个。上下文会加入两侧各两个相邻块,再加上图表、摘要和个人笔记。被选中的模型(本地的 gpt-oss 或 Gemma 4,或托管的 Qwen 或 Opus)写出带引用 [S n] 的答案。每份文档只解析一次:Docling 解析 PDF,Granite-Docling 读取独立公式并与 PDF 的字形核对,文本被切成带标题路径的 300–600 Token 块,并构建关键词和向量索引、图表描述以及覆盖小节、章节和整本书的摘要树。一切都在一块 16 GB 的笔记本 GPU 上运行;只有为某个问题选择托管模型时才会使用它们。

一个问题会经过关键词搜索(BM25)和向量搜索(bge-m3),两个排序列表用倒数排名融合(reciprocal rank fusion)合并,再由一个交叉编码器将前 30 个重排为 6 个。排名最高的命中会带上它们的相邻块,另留出几个槽位加入图表、摘要和我自己的笔记。然后我选择的模型写出带编号引用的答案,任何指向它从未被提供的来源的引用都会在我看到之前被移除。

共有四种回答模式:

快速:在我的 GPU 上运行 gpt-oss-20b。免费,约 17 秒。

均衡:在我的 GPU 上运行 Gemma 4 26B-A4B。免费,约一分钟,还能看图。

Qwen:通过 OpenRouter 调用 Qwen3.8-27B,固定到单一提供商。每问约 0.015 美元。

深度:Claude Opus 5.5。每问约 0.14 美元。

本地是默认选项。托管模型是按每个问题单独选择的,而且应用会明确说明它发送了什么。

在编写检索代码之前,我先根据自己的资料写了 35 道金标准问题:20 道来自四本教材,15 道来自我今年发表的一篇论文。对每一道,我都写下了预期的页码,以及一个好答案必须包含的要点。对那 20 道教材问题,我还列出了必须送达模型的事实。

每个答案都由全新的、此前没见过这项工作的评分智能体评为好、弱或错。多次运行产生的答案被混洗成一个匿名包,不含模型名称,每个答案都附上提供给它的段落。最后这个细节被证明是最有用的:它能让你区分开一个推理糟糕的模型和一个从未见过该事实的模型。

发现 1:教材与论文之分

为了把「小模型」和「本地运行时」区分开来,我用五套配置跑了同样的 35 道题:三个开放模型通过 API 以全精度运行、同一个 gpt-oss 在本地运行,以及作为参照的 Opus。全部使用相同的检索和相同的提示词,并在同一个盲评包中评分。

相同的问题,相同的检索:在 20 道教科书题目和 15 道新论文题目上给出好的回答,一轮盲评。图片由作者提供。

在相同检索条件下对相同问题给出好的回答,并对照事先写好的要点进行盲评。先看教科书题目,再看论文题目:Claude Opus 5.5(API)20/20 到 9/15;Qwen3.8-27B(API)19/20 到 7/15;Gemma 4 26B-A4B(API)16/20 到 1/15;gpt-oss-20b(API)12/20 到 2/15;在我笔记本电脑上运行的 gpt-oss-20b 12/20 到 0/15。青色线条是 Opus 和稠密的 27B Qwen;橙色线条是激活参数量约 4B 的混合专家模型。

在教科书题目上,得分分别是 20、19、16、12 和 12,满分 20。还行。在论文题目上,得分分别是 9、7、1、2 和 0,满分 15。

再读一遍:这些段落就摆在每个模型面前。本地运行的 gpt-oss 拿到了和 Opus 相同的段落,论文题目却依然一道都没答对。

我的解读是:在教科书材料上,小模型已经多次见过这些概念,这些段落只需要提醒它们即可。而在面对一篇新论文时,答案必须完全依据这些段落来构建,小模型正是在这里崩掉。它们会用教科书里通常接着出现的内容来填补空白,而不是用论文实际说的内容。

两个附带结果让这一点更清晰:

量化不是问题。通过 API 以原生精度运行的 gpt-oss 得分与我的本地副本相当(14 对 12 个好回答,符号检验 p = 0.69)。本地经过 QAT 的 Gemma 也与其全精度 API 版本表现持平。在这套硬件上,瓶颈是模型本身,而不是运行时。

一个中等规模的托管模型已经接近前沿水平。Qwen3.8-27B 与 Opus 的差距并不明显(26 对 29),而每个问题的成本约为后者的十分之一。不过它很慢:每个回答大约需要三分钟,其中几乎全部时间都花在思考上。

所以我的应用里的模式建议很简单:查阅你已经熟悉的书籍内容用 Fast,任何新内容用 Deep 或 Qwen。

发现 2:大多数失败都是检索失败

把答案与其对应段落放在一起评分,揭示了另一件事。一次又一次,糟糕的回答之所以出现,是因为事实根本没有到达模型,而不是因为模型推理得差。

最大的提升来自检索,而不是模型。前两行统计的是好的回答;第三行统计的是所有所需事实都被检索到的问题。图片由作者提供。

对检索做了三处改动,每一处都在各自的问题集上做了前后对比测量。Deep 模式下的图表题,在加入图表描述并直接发送图片本身之后:从 10 题中 1 题答对提升到 9 题。Deep 模式下的概览题,在加入摘要树之后:从 8 题中 1 题答对提升到 8 题。所有所需事实均被检索到的书籍题,在把前后各一个相邻分块增加到各两个之后:从 20 题中 16 题提升到 19 题。

图表。我写了 10 个答案只存在于图表中的问题。Deep 答对了 10 题中的 1 题。其中 9 个失败都是检索问题:没有任何段落包含该数值。我让人把每张图表都描述了一遍(397 张图,用 Sonnet 5.5 花费 3.50 美元),这样搜索就能找到它们,而当某张图被检索到时,图片本身也会传给模型。结果:10 题中答对 9 题。

概览类问题。“第12章讲什么,它如何建立在第9章基础上?”没有任何单个分块能承载一章的论证。一个本地模型写出了一棵摘要树(节、章、整篇文档:2.9 小时生成 281 条摘要,成本 $0)。概览类问题从 8 个中对 1 个提升到 8 个中对 8 个。在完整的 16 题集上,16 题表现好,对比 8 题,且没有任何问题是在没有摘要树时表现更好(p = 0.008)。

邻近分块。分块边界常常落在匹配的句子和你需要的公式之间。在排名前三的命中结果两侧各加入两个邻近分块,使所有所需事实均被检索到的书籍问题比例从 16/20 提升到 19/20。

这些都没有改变模型。它们全都改变了模型能看到的内容。

发现 3:即使输入良好,本地模型仍在哪些地方失败

有些任务我努力想留给本地模型,却做不到:

读取图表。我在十张图上对四个描述器做了盲评比较。Gemma 4(开启或关闭思考)和 Qwen3-VL 8B 误读了坐标轴刻度,其中一个偏差约 25 倍,还编造了数值。Qwen3-VL 在十张图中编造了七个数值。Sonnet 5.5 在我自己的盲评中十张里有九张表现良好。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Local LLMs Handle Textbook RAG but Fail on New Research Papers)

I read a lot of technical material: machine learning textbooks, particle physics, and papers in my own field. I wanted an assistant that answers questions about my library, cites the exact page, shows me the passage, and runs on my own laptop. No uploads, no subscription, no guessing where an answer came from.

To be honest about how: I didn't type most of the code myself. Claude Code built it on my machine, phase by phase, while I designed it, wrote the test questions, read every report and decided at every gate. That way of working is a story on its own. This post is about what the measurements showed, because some of it surprised me.

The short version: on textbook questions, small local models do well. On a paper published this year, they collapse to almost zero, with the same retrieval, the same prompt and the same passages in front of them. If you build a local RAG system, how new your material is to the model matters more than how hard the question sounds.

Screenshot of the study assistant in three panels: a library of two papers on the left, a chat in the middle where Claude Opus answers a question about a paper’s notation with numbered citations, and the paper open on page 4 on the right with the cited passages highlighted.

Asking about my own paper: the answer cites its passages, and the viewer highlights them on the page. Image by the author.

The question asks what K, C, T and H mean in my paper on autoregressive sequence models for seismic wavefields, how long one token lasts, and how many tokens a 240 s rollout needs. The hosted Claude Opus answer explains each symbol (K = 16 samples per token, C = 3 channels, T = 320 context tokens, H = 4 predicted tokens), works out one token as about 8.4 s and the rollout as about 28 tokens, and cites passages S1 and S4. The PDF viewer shows the Method section with the retrieved passages highlighted. Below the chat, the search scope is limited to the ticked paper and the four answer modes are shown with their costs.

What I built, in one picture

The app is a FastAPI server with a React front end. You drop PDFs and PowerPoint decks into a folder. Each document is parsed once with Docling, display equations are read again by Granite-Docling and checked against the PDF's own glyphs, and the text is cut into 300–600-token chunks that never split an equation, a table or a code block.

How one question is answered. Everything runs on one 16 GB laptop GPU. Image by the author.

A question (follow-ups are first rewritten by a small 1.7B helper model) goes to two searches: keyword search with BM25 in SQLite FTS5, and meaning search with bge-m3 vectors. The results are merged and a cross-encoder reranks the top 30 down to 6. The context adds two neighbouring chunks on each side plus figures, summaries and personal notes. The chosen model (local gpt-oss or Gemma 4, or hosted Qwen or Opus) writes the answer with citations [S n]. Once per document, Docling parses the PDF, Granite-Docling reads display formulas and checks them against the PDF’s glyphs, the text is cut into 300–600-token chunks with heading paths, and keyword and vector indexes, figure descriptions and a summary tree of sections, chapters and the whole book are built. Everything runs on one 16 GB laptop GPU; hosted models are used only when chosen for a question.

A question goes through keyword search (BM25) and vector search (bge-m3), the two ranked lists are merged with reciprocal rank fusion, and a cross-encoder reranks the top 30 down to 6. The top hits bring their neighbouring chunks, and a few reserved slots add figures, summaries and my own notes. Then the model I chose writes an answer with numbered citations, and any citation to a source it was never given is removed before I see it.

There are four answer modes:

Fast: gpt-oss-20b on my GPU. Free, about 17 s.

Balanced: Gemma 4 26B-A4B on my GPU. Free, about a minute, can look at images.

Qwen: Qwen3.8-27B through OpenRouter, pinned to one provider. About $0.015 a question.

Deep: Claude Opus 5.5. About $0.14 a question.

Local is the default. Hosted is a per-question choice, and the app says exactly what it sends.

Before writing the retrieval code, I wrote 35 gold questions from my own sources: 20 from four textbooks and 15 from a paper of mine published this year. For each I wrote down the expected pages and the key points a good answer must contain. For the 20 textbook questions I also listed the facts that must reach the model.

Every answer was graded good, weak, or wrong by fresh grader agents that hadn't seen the work. Answers from several runs were shuffled into one anonymised pack, without model names, and each answer was shown with the passages it was given. That last detail turned out to be the most useful one: it lets you tell a model that reasoned badly from a model that never saw the fact.

Finding 1: the textbook/paper split

To separate "small model" from "local runtime", I ran the same 35 questions through five setups: three open models via an API at full precision, the same gpt-oss locally, and Opus as the reference. All with the same retrieval and the same prompt, graded in one blind pack.

Same questions, same retrieval: good answers on 20 textbook and 15 new-paper questions, one blind grading round. Image by the author.

Good answers on the same questions with the same retrieval, blind-graded against key points written in advance. Textbook questions, then paper questions: Claude Opus 5.5 (API) 20/20 to 9/15; Qwen3.8-27B (API) 19/20 to 7/15; Gemma 4 26B-A4B (API) 16/20 to 1/15; gpt-oss-20b (API) 12/20 to 2/15; gpt-oss-20b on my laptop 12/20 to 0/15. Teal lines are Opus and the dense 27B Qwen; orange lines are mixture-of-experts models with about 4B active parameters.

On the textbooks, the scores were 20, 19, 16, 12 and 12 out of 20. Fine. On the paper they were 9, 7, 1, 2 and 0 out of 15.

Read that again: the passages were in front of every model. The local gpt-oss got the same passages as Opus and still got zero of the paper questions right.

My reading: on textbook material the small models have seen the ideas many times. The passages only need to remind them. On a new paper, the answer has to be built from the passages alone, and that is where small models fall apart. They fill gaps with what usually comes next in a textbook, not with what the paper says.

Two side results made this cleaner:

Quantisation is not the problem. gpt-oss through an API at native precision scored like my local copy (14 vs 12 good, sign test p = 0.69). The local QAT Gemma matched its full-precision API twin too. On this hardware the limit is the model, not the runtime.

A mid-size hosted model is close to the frontier. Qwen3.8-27B was not significantly behind Opus (26 vs 29), for about a tenth of the price per question. It is slow, though: about three minutes per answer, almost all of it thinking.

So the mode advice in my app is simple: Fast for looking things up in books you know, Deep or Qwen for anything new.

Finding 2: most failures were retrieval failures

Grading answers together with their passages showed something else. Again and again, a bad answer happened because the fact never reached the model, not because the model reasoned badly.

The biggest gains came from retrieval, not from the model. The first two rows count good answers; the third counts questions with every needed fact retrieved. Image by the author.

Three changes to retrieval, each measured before and after on its own question set. Figure questions in Deep mode, after adding figure descriptions and sending the image itself: 1 to 9 good out of 10. Overview questions in Deep mode, after adding a summary tree: 1 to 8 good out of 8. Book questions with every needed fact retrieved, after going from one to two neighbour chunks each way: 16 to 19 out of 20.

Figures. I wrote 10 questions whose answers live only in a plot. Deep got 1 of 10. Nine failures were retrieval: no passage contained the value. I had every figure described once (397 figures, $3.50 with Sonnet 5.5) so search can find them, and the image itself goes to the model when its figure is retrieved. Result: 9 of 10.

Overview questions. "What is chapter 12 about and how does it build on chapter 9?" No single chunk holds a chapter's argument. A local model wrote a summary tree (sections, chapters, whole document: 281 summaries in 2.9 hours, $0). Overview questions went from 1 of 8 to 8 of 8. On the full 16-question set, 16 good against 8, with no question good only without the tree (p = 0.008).

Neighbour chunks. A chunk boundary often falls between the sentence that matches and the equation you need. Adding two neighbours each way around the top three hits took the share of book questions with every needed fact retrieved from 16/20 to 19/20.

None of these changed the model. All of them changed what the model could see.

Finding 3: where local models broke, even with good inputs

Some jobs I tried hard to keep local and couldn't:

Reading figures. I compared four describers blind on ten figures. Gemma 4 (thinking on or off) and Qwen3-VL 8B misread axis scales, one by a factor of about 25, and invented values. Qwen3-VL invented seven values across the ten figures. Sonnet 5.5 got 9 good of 10 in my own blind rating.

Reading web pages. With web search, Deep went from 3–4 to 11 of 11 good on questions the library couldn't answer. Balanced got 2 of 11. Even when the app chose the pages itself and handed Balanced clean excerpts, it was still 2 of 11. It misread the pages: a 5.8σ tension became 8–σ\sigmaσ, a Python version was off by one.

Writing checkable claims. I added a checker: the model writes its derivation's claims in a small block, and SymPy checks each one, units included. Deep wrote checkable claims in 10 of 10 derivations. The local models did in 2 to 6 of 10, and their errors were mostly outside the claims they wrote. Best-of-N resampling on failed checks gave no gain (4 good with or without it).

And one I didn't expect: offering a tool can hurt. Balanced scored 21 good without a "propose a web search" tool and 17 with it, on the same library questions, none of which needed a web search. For small models, every extra tool is another decision they can get wrong.

Finding 4: with 35 questions, know your noise

This part saved me from fooling myself more than once.

Graders drift. The same set of answers graded in two different rounds moved by up to 3 good answers. So I only compare runs inside one blind pack.

Use the sign test. For two runs on the same questions, only the questions where they disagree carry information. The summary tree won 8–0: p = 2.2−82.2^{-8}2.2−8 ≈ 0.008. Real. In my final evaluation, Balanced lost 4 answers and gained 1 against its reference: p = 0.375. Noise.

Use intervals. Deep's 29 of 35 in an early round sounds like "83%". The Wilson 95% interval is 67% to 92%.

Estimate the swing. If each of 32 answers is good with probability about 0.55, the difference between two runs has a standard deviation of (2⋅32⋅0.55⋅0.45)\sqrt{(2·32·0.55·0.45)}(2⋅32⋅0.55⋅0.45)

​ ≈ 4 answers. A drop of 3 is less than one standard deviation.

None of this is advanced statistics. It takes ten minutes to learn, and it changes which results you believe.

What I'd tell you before you build one

Write the test questions first, from your own material, with key points. A test set written after the system tends to test what you built.

Grade with the passages visible. It tells you whether to fix retrieval or change the model.

Split your questions by novelty. Report "familiar" and "new" material separately. A single average hides the most important difference.

Fix retrieval before you buy a bigger model. Neighbours, a reranker, figure descriptions and a summary tree gave me more than any model swap.

Let small models do small jobs. Keep their tool list short. Route figures, web pages and derivations to a stronger model, and say so in the interface.

Fix the bar before the run, and report misses as misses. Several of my gates failed; each was decided openly.

Show the source. Every answer cites its page, and a click opens it. When the model is wrong, I see it in seconds.

The final version answers 34 of my 35 gold questions well in Deep, for about 14 cents a question. Fast answers textbook lookups for free in about 17 seconds. The two together cover most of my daily use, and I always know where an answer came from.

The most useful thing I learned is not in the code. A local model is a good librarian for material that looks like what it was trained on, and an unreliable reader of anything new. Build for that, and measure it.

Thank you for reading. The code is open source under AGPL-3.0: https://github.com/wesmail/study-assistant-share. Try it on your own library, open an issue when something breaks, and send me the questions it gets wrong. That’s how the measurements get better.

How Retrieval Augmented Generation Can Enhance Your LLMs by Integrating a Knowledge Base

August 8, 202312 min read

Have you ever wanted to build an assistant that knows when to talk and when to remain silent? Learn how to do it using open-source models.

January 22, 202416 min read

More efficient language models

October 4, 20239 min read

Your guide to the web's new LLM-ready content standard

November 28, 20245 min read

An introduction to a basic prompt engineering strategy

September 19, 20235 min read

Speeding Up Llama: A Hybrid Approach to Attention Mechanisms

January 10, 202512 min read

Otherwise you may end up with silent mistakes or harsh consequences

October 25, 20235 min read

LLMs are great... if they can fit all of your data.

December 20, 20248 min read

Clarifying the math behind Anthropic’s representation workspace

September 11, 20266 min read

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Towards Data Science(Local LLMs Handle Textbook RAG but Fail on New Research Papers),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗