研讨会回顾:从数据到向量再到上下文
相似度无法告诉你的东西
想象一下,针对某个功能提出的一个问题,它最相近的两个邻居分别是三年前的设计文档和今天早上扫描的代码库。两者都与查询相似。但只有一个描述的是今天实际运行的代码,而相似度分数无法告诉你哪一个才是。
每个时代的检索都解决了前一个时代的局限。关键词搜索基于倒排索引运行,并用相关性指标排序(IDF 可追溯至 1972 年,BM25 至 1994 年),从而在规模化场景下实现精确匹配。但 "cancel" 永远匹配不上 "terminated subscription"。
语义搜索填补了这一空白。借助 word2vec(2013)、用于近似最近邻搜索的 HNSW(2016)和 BERT(2018),检索能够跨越词汇找到相关的含义。它的局限正是设计文档问题。相似并不等于权威。
RAG 源自 Lewis 等人 2020 年的工作,它为模型提供了一种在查询时以知识库为依据来支撑答案的方式,此后几乎出现在每一个聊天演示中。不过这种依据是未经校验的。模型拿到什么就用什么,没有任何步骤去检查它是不是正确结果,或者模型是否真的用了它。
为什么高召回率仍会给出错误答案
检索质量通常用 Recall@K 来衡量。Recall@10 是指相关文档出现在前 10 个结果中的比例。它能很好地衡量信息究竟是否存在于其中。
现在假设有一条策略已被更新的版本取代。检索会同时返回两个版本,而召回率把两者都算作相关。一个基于旧版本、看似合理的答案得分为零,而 Recall@K 看起来却依然正常。对于调用方无权查看的文档,情况也一样:对查询而言相关,对答案而言却越界。
那么,如何让语义搜索更准确?很多时候,搜索本身已经完成了它的工作。合并之后,一家公司的财年从1月到12月,另一家的从2月到次年1月,因此同一个季度在双方对应的是不同月份。对于任何横跨两家公司的问题,语义搜索能找到所有相关文档,却无法说明该如何把它们组合起来。
在这种情况下,调优检索器也无济于事。没有任何相似度分数会带有对齐两套财年的规则。
检索用 Recall@K 来评分:正确的文档是否被召回了?上下文则根据据此做出的决策来评分。目前为止的每个示例都沿同样的方式分野:
第四季度移动业务增长预测
完整数据,包括9月8日的一次峰值
那个9月8日是合作方为期一天的促销活动,已从预测中排除
这个功能是如何工作的?
一份三年前的设计文档和今天早上的代码扫描结果
哪个来源描述了当前运行的代码
合并后涵盖两家公司的任何数字
来自双方的所有相关文件
1月至12月的财年如何与2月至次年1月的财年对应
该政策允许什么?
当前政策及其所取代的版本
哪个版本现行有效
为什么当智能体读取答案时,这一点更重要
一个人如果看到远高于预期的预测,或者看到本不该下降的数据在下降,会挑起眉毛并再次核对。自主智能体则可能不做这项检查就直接把结果发出去,而且智能体调用检索的量级,是人类搜索从未产生过的。
高负载的智能体可能不会注意到结果不合理或已经过时,也可能不会自行对答案进行事实锚定。它缺少能告诉它这些的背景信息。
那为什么不把一切都交给模型,让它自己判断哪些是相关的?因为更多的输入往往会让答案变得更差。
Chroma 的“上下文腐化”(context rot)报告评估了 18 个 LLM,发现随着输入长度增加,性能变得越来越不可靠,即使在刻意设计的简单任务上也是如此。在 LongMemEval 上,平均约 300 Token 的聚焦提示词,表现优于平均约 113,000 Token 的完整提示词。智能体同样如此:把并非针对其任务的信息塞给它,结果就会变差。
问题变了。返回的文档是否正确依然重要,但智能体还需要知道,对于当前这项任务,什么内容是有效的、可信的、被允许的、适合使用的。
决策发生在哪里
搜索依然保留。不同任务需要不同的检索工具,很多任务还需要不止一种。我们把稠密向量、稀疏向量以及带 BM25 排序的全文搜索放在同一个索引里。只要涉及精确匹配,关键词搜索依然有它的价值,而这三者都内建在数据库中,而不是外挂在数据库旁边。
决策层位于搜索之上。每个任务都有一个任务上下文:摘要、实体、策略与规则,以及在任何查询到来之前就在后台整理好的知识制品(knowledge artifacts)。该上下文会判定哪些制品与任务相关,然后按需对索引发起词法、向量或混合查询。
以并购为例。搜索其实已经找到了所有文档。缺的是语义信息:一套财年日历如何映射到另一套,而这部分信息与政策和规则一起存放在任务上下文中。被取代的政策也是同样的道理,databot 的一日促销活动亦然。索引提供数据,而上下文中的规则决定如何处理这些数据。
我们打造了 Pinecone Nexus——面向智能体的知识引擎,就是为了充当这一层。Nexus 的策展功能会维护一份已处理来源的账本,以内容哈希进行跟踪,因此每次运行只处理发生变化的内容。答案会附带引用,标注每条论断背后的文件、页码和原文片段。
优先级规则也会被明确写下来。上下文的默认指令相当于针对该上下文的每一次查询的系统提示词,它们可以告诉模型如何处理相互冲突的来源,例如哪个版本的政策优先。
我们认为 Nexus 并非适用于所有工作负载。如果你的检索模式很简单,你相当确定前 10 条结果里就有答案,没有复杂的规则或政策来决定哪个来源优先,而且你愿意自己承担接地(grounding)工作,那么 RAG 就足够了。
像大型在线商店的每一笔购买这类每秒钟都在变化的运营数据,大概需要实时、动态的视图。实时数据流和全天就地编辑的记录,目前都不太适合 Nexus。它最适合数据变化节奏较慢的场景,比如合同和研究。
在你花一个冲刺周期去做嵌入向量或重排序之前,先检查一下检索是否已经找到了文档。如果已经找到,那么问题就出在负责决策的那一层。完整回放见 YouTube。要构建这一层,可以从 Nexus 文档入手,或申请试用。
这篇文章对你有帮助吗?
今天就开始构建有知识的 AI
免费创建你的第一个索引,当你准备扩展规模时,再按用量付费。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Webinar Recap: From Data to Vectors to Context)
What Similarity Can't Tell You
Imagine the two nearest neighbors for a question about a feature are its design doc from three years ago and a scan of the codebase from this morning. Both are similar to the query. Only one describes the code that runs today, and a similarity score can't tell you which.
Each era of retrieval fixed the limit of the one before it. Keyword search runs on an inverted index and ranks with relevance metrics (IDF dates to 1972, BM25 to 1994), which gives exact match at scale. But "cancel" never matches "terminated subscription."
Semantic search closed that gap. With word2vec (2013), HNSW for approximate nearest-neighbor search (2016) and BERT (2018), retrieval finds related meaning across vocabulary. Its limit is the design-doc problem. Similar isn't authoritative.
RAG, from Lewis et al. in 2020, gave models a way to ground answers in a knowledge base at query time, and it has been in nearly every chat demo since. The grounding is unchecked, though. The model gets whatever comes back, with no step that checks whether it is the right result or whether the model used it.
Why Good Recall Still Gives Wrong Answers
Retrieval quality is usually measured as recall at K. Recall@10 is the share of relevant documents that land in the top 10 results. It's a good measure of whether the information is there at all.
Now take a policy that a newer one has superseded. Retrieval returns both versions, and recall counts both as relevant. A plausible answer grounded in the old version scores zero, and Recall@K can look fine anyway. The same goes for a document the caller isn't allowed to see: relevant to the query, and out of bounds for the answer.
So how do you make semantic search more accurate? Often the search is already doing its job. After a merger, one company's fiscal year runs January to December and the other's runs February to January, so the same quarter covers different months on each side. For any question that spans both, semantic search finds every relevant document. It can't say how to combine them.
Tuning the retriever won't help there. No similarity score carries a rule for lining up two fiscal calendars.
Retrieval is scored on Recall@K: did the right documents come back? Context is scored on the decision made with them. Every example so far splits the same way:
Projected Q4 mobile growth
Complete data, including a September 8 spike
That September 8 was a one-day partner promotion, excluded from the forecast
How does this feature work?
A three-year-old design doc and this morning's code scan
Which source describes the code that runs today
Any figure spanning both companies after a merger
Every relevant document from both
How a January-December fiscal year lines up with a February-January one
What does the policy allow?
The current policy and the version it superseded
Which version is in force
Why It Matters More When an Agent Reads the Answer
A person who sees a forecast far too high, or dropping when it shouldn't, raises an eyebrow and double-checks. An autonomous agent might ship it without that check, and agents call retrieval at volumes no human search ever generated.
A high-volume agent may not notice that results don't make sense or are out of date, and it may not ground the answer on its own. It's missing the context that would tell it.
So why not give the model everything and let it work out what's relevant? Because more input tends to make answers worse.
Chroma's context rot report evaluated 18 LLMs and found that performance grows less reliable as input length grows, even on deliberately simple tasks. On LongMemEval, focused prompts averaging about 300 tokens beat full prompts averaging around 113,000 tokens. The same holds for agents: load one with information that isn't specific to its task and the outcomes get worse.
The question changes. Whether the right documents came back still matters, but an agent also needs to know what is valid, trusted, permitted and appropriate to use for this task.
Where the Deciding Happens
Search stays. Different tasks need different retrieval tools, and plenty need more than one. We put dense vectors, sparse vectors and full-text search with BM25 ranking in one index. Keyword search still earns its place wherever exact match matters, and all three are built into the database rather than beside it.
The deciding layer sits on top of search. Each task gets a task context: summaries, entities, policies and rules, plus knowledge artifacts curated in the background before any query arrives. The context resolves which artifacts are relevant to the task, then issues lexical, vector or hybrid queries against the index as it needs them.
Take the merger. Search already found every document. The missing piece is semantic information: how one fiscal calendar maps onto the other, held in the task context next to the policies and rules. The superseded policy works the same way, and so does the databot's one-day promotion. The index supplies the data, and a rule in the context decides how to treat it.
We built Pinecone Nexus, our knowledge engine for agents, to be that layer. Nexus curation keeps a ledger of processed sources, tracked by content hash, so a run only touches what changed. Answers carry citations to the file, page and verbatim span behind each claim.
Precedence gets written down, too. A context's default instructions act as a system prompt for every query against the context, and they can tell the model how to handle conflicting sources, such as which policy version wins.
We don't think Nexus is the answer for every workload. If your retrieval patterns are simple, you're fairly sure the top 10 results hold the answer, no complex rules or policies decide which source wins, and you're fine owning the grounding, RAG is enough.
Operational data that changes every second, like each purchase on a large online store, probably wants a live, dynamic view. Real-time feeds and records edited in place all day aren't a good fit for Nexus today. It works best where data changes on a slower schedule, such as contracts and research.
Before you spend a sprint on embeddings or reranking, check whether retrieval already found the documents. If it did, the fix belongs to the layer that decides. The full replay is on YouTube. To build that layer, start with the Nexus docs or request a trial.
Was this article helpful?
Start building knowledgeable AI today
Create your first index for free, then pay as you go when you're ready to scale.
原文为英文,正文由自动抓取并保留原文。
原文发布于 Pinecone Blog(Webinar Recap: From Data to Vectors to Context),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。