斯坦福开源 Memorilla:面向冻结 LLM 的潜在记忆模块
斯坦福大学研究人员发布了 Memorilla,这是一个面向 LLM 的开源记忆系统。Memorilla 记忆系统会读取整个文档集合,然后只交给一个冻结的 LLM 16 个记忆 Token。在 FactKG 上,冻结的 Qwen3-8B 使用 Memorilla 得分 84.33%,而将所有文档放入上下文的方式得分为 80.43%。实现这一结果时,其文档 Token 用量减少了 108 倍。代码、训练配方和检查点均以 Apache 2.0 协议公开。你可以通过我们的 GPULeaderboard.com chrome 扩展在这里查看部署选项和规格。
Memorilla 是一个记忆模块,可将文档集合转化为 16 个向量,供冻结的 LLM 使用。该成果来自 Jure Leskovec 的 SNAP 团队,共同作者来自多伦多大学、宾夕法尼亚大学和 MIT。
冻结的 LLM 只能利用其上下文窗口中的内容。全上下文提示会随着历史增长而变慢。检索则把与问题最相似的片段拼接进来,可能漏掉较弱的线索。Memorilla 走了第三条路:它学习每个问题需要从整个存储中获取什么。
该流程分为 3 个步骤:
只有记忆模块参与训练:对于 8B 解码器,其参数量为 194,123,776。训练分 3 个阶段进行。第 1 阶段使用约 198 万条 Wikipedia 释义重构样本。第 2 阶段加入来自 10 个问答与摘要数据集的约 134 万条样本。第 3 阶段在全部 8 个基准测试上微调。完整配方在 4 块 H100 GPU 上耗时约 19 小时。
根据论文的主表,所有基线都使用同一个冻结的 Qwen3-8B。分数由研究团队自行报告。
Memorilla 在 3 个基准测试上落败。在 PubMedQA 上它得分 59.00%,而 LLMLingua-2 达到 69.00%。这些摘要总共只有 324 个 Token,因此压缩几乎没有发挥空间。MemoRAG 在 PersonalizationV4(67.88% 对 66.65%)和 NarrativeQA ROUGE-L(20.51 对 19.31)上领先。在这两项中,MemoRAG 的文档 Token 用量分别高出约 690 倍和 50 倍。
在 NarrativeQA 上压缩达到峰值:16 个 Token 替代 67,385 个文档 Token,压缩比达 4,212 倍。作为智能体记忆时,8 个记忆 Token 将 Qwen3-4B TextWorld 智能体的平均成功率从 18.6% 提升到 24.2%。
在 1 块 H100 上,Memorilla 在 512 篇文档(约 128K Token)时用 0.74 秒给出答案。全上下文方案则从 0.75 秒放慢到 19.9 秒,慢了 27 倍。记忆模块本身耗时约 2 毫秒。
可选的记忆缓存会预计算交叉注意力的键和值。在 1 块 B200 上,一个问题在 100 万篇文档时耗时 8.1 毫秒,在 400 万篇时耗时 33 毫秒。使用 torch.compile 后,100 万篇的场景降至 5.0 毫秒。该缓存在 400 万篇文档时占用 80 GiB。
在 Linux 上用 uv 安装,然后加载 MemoryVLLM.from_pretrained。首次运行会下载约 25 GB 的模型。检查点位于 memorilla/Memorilla-Qwen3-8B,分为 Stage 1、2、3、PV4 和 PMv2 文件夹。
每个检查点都必须搭配其训练时所用的解码器运行(见评估文档)。自定义数据需要包含 collection_id、question、answer 和 documents 列的 parquet 文件。研究团队还发布了 PersonalizationV4:149 个合成用户,每人 200 段对话,采用 CC BY 4.0 许可。
基准测试分数来自 Memorilla 的论文,其中所有基线都在同一个 Qwen3-8B 上运行。许可证信息来自各项目的代码仓库。
查看论文、浏览 GitHub 仓库并下载检查点。你可以通过我们的 GPULeaderboard.com Chrome 扩展在这里查看部署选项和规格。
需要与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布或线上研讨会等?欢迎联系我们。
文章《Stanford 研究团队开源 Memorilla:面向冻结大语言模型的潜在记忆模块》首发于 MarkTechPost。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs)
Stanford researchers have released Memorilla, an open-source memory system for LLMs. The Memorilla memory system reads a whole document collection and hands a frozen LLM just 16 memory tokens. On FactKG, a frozen Qwen3-8B scored 84.33% with Memorilla, against 80.43% with every document in context. It did that with 108x fewer document tokens. Code, training recipes and checkpoints are public under Apache 2.0. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.
TL;DR
- Size: 194M-parameter trainable memory module. It always outputs 16 memory tokens, whatever the collection size. The encoder (Qwen3-Embedding-4B) and decoder (Qwen3-8B) stay frozen.
- Runs on: Linux with CUDA GPUs and Python 3.11. The quickstart needs 1 GPU with 80 GB (recipes ran on H100 80GB). No quantized variants disclosed.
- Performance: Highest reported average across 8 benchmarks (49.68), top score on 5 of 8.
- Best: PersonaMem-v2 accuracy of 44.60%, versus 27.92% for full context.
- Worst: PubMedQA accuracy of 59.00%, versus 69.00% for LLMLingua-2.
What is Memorilla?
Memorilla is a memory module that turns a document collection into 16 vectors for a frozen LLM. The work comes from Jure Leskovec’s SNAP group, with co-authors from Toronto, Penn and MIT.
A frozen LLM can only use what sits in its context window. Full-context prompting grows slower as history grows. Retrieval pastes the chunks most similar to the question, and can miss weaker clues. Memorilla takes a third route: it learns what each question needs from the whole store.
How does Memorilla work?
The pipeline has 3 steps:
- Encode once. A frozen Qwen3-Embedding-4B embeds each document a single time. New documents never force re-encoding of old ones.
- Read per question. 16 learned slots attend to the question and the full store. The module uses 2 cross-attention layers with 8 heads.
- Inject. The slots are projected into Qwen3-8B’s input embedding space and prepended to the question.
Only the memory module trains: 194,123,776 parameters for the 8B decoder. Training runs in 3 stages. Stage 1 uses about 1.98M Wikipedia paraphrase-reconstruction examples. Stage 2 adds about 1.34M examples from 10 QA and summarization datasets. Stage 3 fine-tunes on all 8 benchmarks. The full recipe takes about 19 hours on 4 H100 GPUs.
How does Memorilla perform on benchmarks?
All baselines use the same frozen Qwen3-8B, per the paper’s main table. Scores are self-reported by the research team.
- FactKG: 84.33% accuracy, versus 80.43% full context and 80.38% MemoRAG.
- PersonaMem-v2: 44.60%, versus 30.90% for xRAG, the next best method.
- TriviaQA: 57.00 exact match, just ahead of MemoRAG at 56.60.
- LaMP-4 / LaMP-7: 18.15 and 48.37 ROUGE-L, both the best scores.
- Average: 49.68, versus 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0.
Memorilla loses on 3 benchmarks. On PubMedQA it scores 59.00%, while LLMLingua-2 reaches 69.00%. Those abstracts total only 324 tokens, so compression has little to gain. MemoRAG leads on PersonalizationV4 (67.88% vs 66.65%) and NarrativeQA ROUGE-L (20.51 vs 19.31). MemoRAG used about 690x and 50x more document tokens there.
Compression peaks on NarrativeQA: 16 tokens replace 67,385 document tokens, a 4,212x cut. As agent memory, 8 memory tokens lift a Qwen3-4B TextWorld agent from 18.6% to 24.2% average success.
How fast is Memorilla compared to full context?
On 1 H100, Memorilla answers in 0.74 s at 512 documents (about 128K tokens). Full context slows from 0.75 s to 19.9 s, which is 27x slower. The memory module itself takes about 2 ms.
An optional memory cache precomputes cross-attention keys and values. On 1 B200, a question then takes 8.1 ms at 1M documents and 33 ms at 4M. With torch.compile, the 1M case drops to 5.0 ms. The cache costs 80 GiB at 4M documents.
How do you run Memorilla?
Install with uv on Linux, then load MemoryVLLM.from_pretrained. The first run downloads about 25 GB of models. Checkpoints live at memorilla/Memorilla-Qwen3-8B as Stage 1, 2, 3, PV4 and PMv2 folders.
Each checkpoint must run with the decoder it was trained with (evaluation docs). Custom data needs parquet files with collection_id, question, answer and documents columns. The research team also released PersonalizationV4: 149 synthetic users, 200 conversations each, under CC BY 4.0.
Memorilla vs Mem0, MemoRAG and xRAG
Benchmark scores come from Memorilla’s paper, which ran every baseline on the same Qwen3-8B. Licenses come from each project’s repository.
| Memorilla (paper) | MemoRAG (repo) | Mem0 (repo) | xRAG (repo) | |
|---|---|---|---|---|
| Approach | Learned latent memory tokens | Memory model guides retrieval | Extracted text memories | 1-token RAG compression (paper) |
| Doc tokens per query (geo. mean) | 16 | 784 | 212 | 1 |
| What the LLM reads | 16 soft memory embeddings | Retrieved text | Retrieved memory text | 1 projected embedding |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Not disclosed |
| FactKG (Acc) | 84.33 | 80.38 | 72.79 | 78.98 |
| PersonaMem-v2 (Acc) | 44.60 | 28.76 | 27.48 | 30.90 |
| PubMedQA (Acc) | 59.00 | 67.00 | 24.00 | 36.00 |
| NarrativeQA (ROUGE-L) | 19.31 | 20.51 | 13.80 | 16.92 |
| Average, 8 benchmarks | 49.68 | 47.52 (our calc.) | 38.02 | 41.92 |
Key Takeaways
- 16 memory tokens replace whole document collections for a frozen Qwen3-8B.
- FactKG: 84.33% vs 80.43% full context, with 108x fewer tokens.
- Highest reported 8-benchmark average: 49.68.
- Weak spot: PubMedQA at 59.00% and fine-grained facts.
- Apache 2.0 code; 194M-parameter checkpoints on Hugging Face.
Check out the paper, browse the GitHub repo, and download the checkpoints. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.
Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。