当向量成为实现细节
Documents API:原生向量、全文搜索,如今又集成了嵌入能力,更多功能还在路上。
我在 2024 年初加入 Pinecone 时,最让我印象深刻的,是用户在使用 Pinecone 之前必须在 Pinecone 之外做多少工作。好的搜索依赖于提前准备好数据,并编排好应用接收到的查询。过去两年半里,我反复回到同一个问题:如何让 Pinecone 承担更多这类工作,让用户能把更多时间花在构建自己的应用上,而不是花在构建检索流水线上。
向量是这项准备工作中的重要部分,而在我看来,它始终像是一个实现细节。向量对于数据库找到你要找的东西至关重要,但它不该是必须由你自己来构建的东西。
从本质上说,这是一个关于 API 的故事,所以从头讲起是合理的。对我们而言,起点就是 Vectors API。
自带向量(约 2021 年)
Pinecone 最初是一个纯向量数据库;虽然我们现在把最初的 API 称为 Vectors API,但当时我们就叫它“那个 API”。不出所料,它说的都是向量的语言:要么接收向量并存储,要么接收一个查询向量来搜索向量空间。这意味着要把第三方 embedding 提供方接入你的数据流水线,随之而来的是一系列常见的额外工作:身份认证与密钥轮换、速率限制与退避重试、计费,以及监控。大多数应用还分成两部分,一条路径用于加载数据,另一条用于查询,于是你最终得把集成做两遍,其中还要在低延迟查询与高吞吐数据加载之间做权衡。
Vectors API 还要求你让数据符合其请求结构:一个 ID、一个向量,以及按规定格式提供的一些元数据。而且你还面临一个决定:每个向量背后的源文本保存在哪里:要么保存在单独的数据存储中,要么放在 Pinecone 的元数据字段里,这是在应用复杂性与索引成本和性能之间的权衡。
单独的数据存储意味着每个搜索结果都需要反向查找才能找到对应文本。把文本放进元数据字段更简单,但元数据本意是用于你筛选的小值,比如类别。Pinecone 会将其索引为 roaring bitmaps,因此你存储在其中的每一段文本都会增加存储和写入成本,并占用每条记录的元数据限额的一部分。而且这并不会让其中的词变得更容易被搜索。
使用 Vectors API 时,你需要负责嵌入集成,而且通常不得不构建两次(一次用于写入,再一次用于查询),还需要决定相关文本存储在哪里,其中涉及各种权衡。
奠定基础(2024–2025)
很快就能明显看出,我们的每一位用户都在构建同一条嵌入流水线(往往构建两次)。但在我们能够承担其中任何工作之前,必须先有两个基础组件到位。
第一项是推理栈。我们从 2024 年开始自己托管模型,并新增了独立的 embed 和 rerank 端点。这意味着你可以只用一个 API key、一个提供商,就开始简化应用中的部分环节。每次写入和每次搜索之前,你仍然需要自己调用模型,但由于模型现在就在数据库旁边运行、且共用同一个 API key,这为把它们直接与其所支持的数据库工作负载结合起来打开了大门。
第二项是数据库本身更灵活的存储架构。2025 年初,我们发布了 slab 架构:一种面向无服务器索引的全新日志结构架构*。在 slab 架构下,写入先落入内存缓冲区,再刷写到我们称为“slabs”的不可变文件集合,当有必要时,后台压缩会自动把小的 slab 合并成更大的 slab。每个 slab 都是自描述的:它携带自己的数据、自己的索引,以及一条说明自己持有哪种索引的记录。重要的是,这使得索引选择对向量之外的不同类型索引都变得可组合,全文搜索后来也成为其中之一。
🤯 题外话:2025 年我们投入了大量时间,把 slab 架构推广到我们的全部客户群,以便将线上索引从旧的集群架构迁移到新的 slab 架构。在整个迁移过程中,客户不仅能继续读取,还能继续写入。
这就像不只是在飞行途中造飞机,还要在半空中把乘客从旧飞机转移到新飞机上!
想进一步了解我们的架构,可以查看 Pinecone 的工作原理以及我们的 slab 架构指南。
Records API:验证这一模式(2024)
在重建数据库架构的过程中,我们首次尝试通过 Records API 实现集成嵌入(embedding),该 API 于 2024 年底发布。我们随后不久就撰文介绍了它,包括我们如何通过把对延迟敏感的查询嵌入流量与较慢的批量 upsert 工作负载分离来提升其速度。
使用 Records API 时,你在创建索引时指定模型,之后就可以 upsert 文本记录并用文本进行搜索。Pinecone 在写入时对文本做嵌入,在查询时对查询做嵌入,使用的是索引上配置的模型。这验证了我们想要的基本请求模式。用户只需提供文本,嵌入在 Pinecone 内部完成,用户永远不必接触向量。
👉 有趣的是:我们原本预期 Records API 会发展成一个“documents” API,所以实际上把它背后的新 Kubernetes 服务命名为 ‘docs-api’。
但归根结底,Records API 还是太超前了:它的形态无法支撑我们持续构建。在底层,它封装了我们的 Vectors API,文本仍然存放在元数据字段中,因此仍要承担与以往相同的取舍成本。每个索引都绑定到一个模型和一个文本字段,也不给你避免存储文本的选项。由于它直接构建在 Vectors API 之上,Records API 没有空间去演进以支持其他类型的搜索,比如原生全文搜索。
全文搜索改变了形态(2025–2026)
我们早就知道,除了向量搜索之外,还需要全文搜索。它在精确字符串匹配上更胜一筹,比如产品编号和错误信息,而把两者结合起来威力很大。但全文搜索需要文本本身被原生文本搜索引擎分词并建立索引。借助 slab 架构,我们理论上已经为一种新的索引留出了空间,但这一点尚未得到验证。我们当时的 API 也没有为它预留位置。要让全文搜索成为现实,就意味着这两方面都要解决。
于是在 2025 年 11 月下旬,我们把工程团队召集到纽约,把自己关进一间会议室,直到我们有了一块没人敢擦掉的白板,并且清楚知道需要构建什么,才从里面出来。最终归结为三个重要的新组件:
在 slab 架构中原生支持全文搜索
一套灵活的 schema 语言,用于驱动索引配置和查询编排
一套真正的文档 API 和灵活的搜索 API,让我们可以在此基础上持续构建
我们已经撰文介绍了全文搜索的工作原理、其背后的架构,以及它的正式可用。简而言之,文档索引支持用户自定义的带类型字段 schema,首个版本包括:用于全文搜索的字符串字段、用于原生向量搜索的稠密和稀疏向量字段,以及用于过滤的元数据。而搜索 API 让你可以选择文档的评分方式:基于文本字段的关键词相关性,或基于向量字段的语义。
除了在关键词密集的使用场景中提升检索质量之外,全文搜索还解决了「把文本当作元数据存储」的问题。元数据索引的构建目的就是按精确值过滤,而原始文本几乎从来无法从精确值匹配中获益。全文字段的存储方式不同:Pinecone 把原始文本在 slab 中只存一份,并在其旁边构建一套 Tantivy 倒排索引,将每个词项映射到它出现的文档及位置。该索引使用 BM25 为搜索结果打分,还支持 $match_phrase 等强大的文本匹配过滤器,因此任何查询(包括向量搜索)都可以要求文档包含某个特定词或短语。
Documents API 面向一套完全从零开始、自上而下构建的全新索引机制,由文档 schema 驱动,原生全文搜索与向量搜索在同一套 slab 架构中并排放置。它还颠倒了请求的形态:Vectors API 要求你让数据符合它的形态,而 Documents API 的设计则是去适配你的形态。你用自己的字段名写入文档,schema 则规定每个字段应如何被索引。
文档索引和 Documents API 的首个版本聚焦于全文搜索与原生向量。但集成嵌入本来就在计划之中。Documents API 在设计之初就为接纳它以及后续的演进做好了准备,以支持混合搜索、集成重排序、多向量字段等更多能力。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(When vectors became an implementation detail)
The Documents API: native vectors, full-text search, and now integrated embedding, with more on the way.
When I joined Pinecone in early 2024, what stood out was how much work our users had to do outside of Pinecone before they could use us. Good search depends on preparing your data ahead of time and orchestrating the queries your application receives. Over the last 2.5 years I’ve kept coming back to the same problem: how to get Pinecone to take on more of that work, so our users can spend more time building their application and less time building retrieval pipelines.
Vectors are a big part of that preparation, and they always seemed to me like an implementation detail. They're essential to how the database finds what you're looking for, but not something you should have to build yourself.
At its heart, this is an API story, so it makes sense to start at the beginning. For us, that’s the Vectors API.
Bring your own vectors (circa 2021)
Pinecone started as a pure vector database, and while we may now call the original API the Vectors API, we used to just call it “the API”. Not surprisingly, it speaks in terms of vectors: it either takes in vectors and stores them, or takes a query vector to search the vector space with. That means wiring a third-party embedding provider into your data pipeline, with all of the usual extra work that comes with it: authentication and key rotation, rate limits and retry with backoff, billing, and monitoring. Most applications are also split in two parts, with one path for loading data and another for queries, so you end up having to build the integration twice, including balancing the need for low-latency queries with high-throughput data loading.
The Vectors API also asks you to conform your data to its request shape: an ID, a vector, and some metadata in a prescribed format. And you're faced with a decision about where to keep the source text behind each vector: either you need to keep it in a separate data store, or you put it in a metadata field in Pinecone, a tradeoff between complexity in your application and cost and performance in your index.
A separate data store means that every search result requires a reverse lookup to find its corresponding text. Putting the text in a metadata field is simpler, but metadata is meant for small values you filter on, like categories. Pinecone indexes it into roaring bitmaps, so every block of text you store there adds to your storage and write costs and uses up part of the metadata limit per record. And it doesn’t make the words in it any more searchable.
With the Vectors API, you’re responsible for the embedding integration and often have to build it twice (once for writes and again for queries) and need to decide where to store the related text, with trade-offs involved.
Laying the groundwork (2024–2025)
It quickly became clear that every one of our users was building the same embedding pipeline (often twice). But before we could take on any of that work, two fundamental components had to be in place.
The first was an inference stack. We started hosting models ourselves in 2024, adding standalone embed and rerank endpoints. This meant you could use one API key and one provider and begin to simplify some parts of your application. You still had to call the model yourself before every write and every search, but because the models now ran next to the database and under the same API key, that opened the door to combining them directly with the database workloads they supported.
The second was a more flexible storage architecture for the database itself. In early 2025 we shipped the slab architecture: a new log-structured architecture for serverless indexes*. With slab, writes land in an in-memory buffer and flush to immutable sets of files we call “slabs”, and background compaction automatically merges small slabs into larger ones when warranted. Each slab is self-describing: it carries its own data, its own index, and a record of what kind of index it holds. Importantly, that made index selection composable for different kinds of indexes beyond just vectors, and full-text search would later become one of them.
🤯 Side note: We invested a lot of time in 2025 in rolling out the slab architecture to our entire customer base, so that we could migrate live indexes from our older clustering architecture to the new slab architecture. Not only could customers keep reading, but they could also keep writing throughout the migration process.
Akin to not just building an airplane while flying, but also transferring the passengers from the old plane to the new one in mid-air!
To learn more about our architecture, see how Pinecone works and our guide to the slab architecture.
The Records API: proving the pattern (2024)
While we were rebuilding the database architecture, we took our first crack at integrated embedding via the Records API, which launched in late 2024. We wrote about it shortly after, including how we sped it up by separating latency-sensitive query embedding traffic from slower batch upsert workloads.
With the Records API, you specify a model when creating an index, and then upsert text records and search with text. Pinecone embeds the text on the way in and the query on the way out, using the model configured on the index. This proved the basic request pattern that we wanted. Users just bring text, the embedding happens inside Pinecone, and users never have to see a vector.
👉 Fun fact: we expected the Records API to grow into a “documents” API, so we actually named the new Kubernetes service behind it ‘docs-api’.
But ultimately the Records API was premature: it wasn't a shape we could keep building on. Under the hood, it wraps our Vectors API, and the text still lives in a metadata field, with the same tradeoff costs as before. Each index was tied to one model and one text field, and didn’t give you the option to avoid storing the text. Because it was built directly on the Vectors API, the Records API had no room to evolve to support other kinds of search, like native full-text.
Full-text search changes the shape (2025–2026)
We’d known for a long time that we needed full-text search alongside vector search. It wins on exact string matching, like product codes and error messages, and combining the two is powerful. But full-text search needs the text itself tokenized and indexed by a native text search engine. With the slab architecture we had theoretically made room for a new kind of index, but it was unproven. Our APIs at the time also had no place for it. Making full-text search a reality would mean addressing both.
So in late November 2025 we brought the engineering team together in NYC, locked ourselves in a conference room, and didn’t emerge until we had a whiteboard nobody dared to erase and a clear picture of what we needed to build. It came down to three significant new pieces:
Native full-text search to be supported in our slab architecture
A flexible schema language to drive indexing configuration and query orchestration
A true document API and flexible search API we can continue to build on
We've written about how full-text search works, the architecture behind it, and its general availability. In short, document indexes support a user-defined schema of typed fields, which at the first release were: string fields for full-text search, dense and sparse vector fields for native vector search, and metadata for filtering. And the search API lets you choose how documents are scored: by keyword relevance on text fields, or by semantics on a vector field.
Besides improving retrieval quality in keyword-heavy use cases, full-text search fixes the text-as-metadata storage problem. Metadata indexing is built to filter on exact values, and raw text almost never benefits from exact value-matching. A full-text field is stored differently: Pinecone keeps the raw text once in the slab and builds a Tantivy inverted index next to it, which maps each term to the documents and positions where it appears. The index scores search results with BM25, and it also supports powerful text-match filters like $match_phrase, so any query, including a vector search, can require that a document contain a specific word or phrase.
The Documents API fronts an entirely new index mechanism built from the top down, driven by a document schema, with native full-text search and vector search side by side in the same slab architecture. It also inverts the request shape: the Vectors API asks you to conform your data to its shape, while the Documents API is designed to conform to yours. You write documents with your own field names, and the schema says how each field should be indexed.
The first release of document indexes and the Documents API focused on full-text search and native vectors. But integrated embedding was always part of the plan. The Documents API was designed up front to accommodate it and further evolution to support things like hybrid search, integrated reranking, multiple vector fields, and more.
Integrated embedding as a schema field (2026)
You can now configure both integrated embedding and full-text search on a string field:
pc.indexes.create(
name="space-missions",
schema={
"fields": {
"body": {
"type": "string",
"full_text_search": {"stemming": True, "stop_words": True},
"embed": {"model": "llama-text-embed-v2"},
}
}
}
)
Create a document index with one string field that has both full-text search and integrated embedding.
When you write a document, you send plain text. Pinecone embeds it with the field's model (specified by the schema), stores the vector next to the text, and indexes the text for full-text search—all in the same write. If embedding fails, nothing in the request is written. When you search, you send the query as text, and Pinecone embeds it with the same model. The goal is that loading data should feel less like building a data pipeline and more like configuring one.
With integrated embedding, one string field holds the text, its full-text index, and its vector. Both kinds of search can run against it.
Integrated embedding also makes search faster. When you embed the query yourself, every search takes two round trips: one to an embedding API and another to Pinecone with the vector. With integrated embedding it's one call. In our testing, a search with text in one call was about 3.5 times faster at the median than calling our standalone embedding API and then searching with the vector. Most of the difference comes from the extra network round trip that the single call avoids.
Here’s how the Documents API compares to the APIs that came before it:
JSON documents with your own fields
ID, text field and metadata, in JSONL format
Pinecone, per field in the schema
Pinecone, one model per index
Separate store or metadata
Full-text, vector, or text (embedded)
Here’s an example index with a handful of short descriptions of space missions in a single body field, with both search modes enabled. Each document also carries a year and whether the mission was crewed. The schema doesn’t declare those fields, so they’re stored as metadata you can filter on.
index = pc.index("space-missions")
index.documents.upsert(
namespace="missions",
documents=[
{"_id": "voskhod-2", "body": "Voskhod 2, March 1965. A cosmonaut left the capsule through an inflatable airlock and floated outside for about 12 minutes, then had to let air out of his swollen suit to squeeze back in.", "year": 1965, "crewed": True},
{"_id": "apollo-11", "body": "Apollo 11, July 1969. Neil Armstrong and Buzz Aldrin landed in the Sea of Tranquility while Michael Collins orbited above in the command module.", "year": 1969, "crewed": True},
{"_id": "apollo-soyuz", "body": "Apollo-Soyuz, July 1975. An American Apollo spacecraft docked with a Soviet Soyuz in orbit, and the two crews shook hands through the hatch.", "year": 1975, "crewed": True},
{"_id": "voyager-1", "body": "Voyager 1, launched September 1977. It flew past Jupiter and Saturn, and in 2012 became the first spacecraft to reach interstellar space.", "year": 1977, "crewed": False},
{"_id": "sts-61", "body": "STS-61, December 1993. Astronauts on the shuttle Endeavour installed corrective optics in the Hubble Space Telescope to fix its flawed primary mirror.", "year": 1993, "crewed": True},
...
],
)
Upsert documents as plain text, with year and crewed stored as filterable metadata.
Some searches are for an exact string. Someone who types a mission code into a search box wants to find exact references to the mission, and full-text search matches the code itself.
results = index.documents.search(
namespace="missions",
score_by=[{"type": "text", "field": "body",
"query": "STS-61"}],
top_k=2,
include_fields=["*"],
)
== search-text results:
[3.3801] sts-61
STS-61, December 1993. Astronauts on the shuttle Endeavour installed
corrective optics in the Hubble Space Telescope to fix its flawed primary
mirror.
Search the body field for the exact mission code.
Other searches are questions, asked in the searcher's own words. The search API embeds the question with the same model that embedded the documents, and matches on meaning. Since the question is about a person, this search also filters on the crewed field, which leaves out uncrewed missions like Voyager 1.
results = index.documents.search(
namespace="missions",
score_by=[{"type": "embed", "field": "body",
"query": "Who was the first person to float outside a spacecraft?"}],
filter={"crewed": {"$eq": True}},
top_k=2,
include_fields=["*"],
)
== search-embed results:
[0.4294] voskhod-2
Voskhod 2, March 1965. A cosmonaut left the capsule through an inflatable
airlock and floated outside for about 12 minutes, then had to let air out of
his swollen suit to squeeze back in.
[0.2319] apollo-soyuz
Apollo-Soyuz, July 1975. An American Apollo spacecraft docked with a Soviet
Soyuz in orbit, and the two crews shook hands through the hatch.
Search the same field by meaning, using only crewed missions.
Both searches run against the same field in the same index, and neither one requires the client application to handle a vector.
When a search arrives with an embed clause, the Documents API embeds the query by calling our embedding service, with the model the field declares. Our embedding client retries with backoff and spreads load across healthy endpoints, where “healthy” means both a low error rate and low latency (learn more about our load balancing). Query embeddings go to the embedding service as their own small requests and are isolated from the large batches used for writes, so a latency-sensitive search doesn't wait behind someone else's bulk upsert.
The query router reads the index’s manifest, which is usually cached, to find its slabs, then sends the search directly to the executors that hold them. Every executor keeps the index portion of its slabs on local NVMe SSD and in memory, and new slabs are warmed on the executors before being added to the manifest, so searches never get stuck waiting on a cold slab.
A search with integrated embedding: the Documents API embeds the query in the same region, then the executors score slabs that are already cached on local SSD and in memory.
Bottom line, your search starts executing as soon as the query embedding comes back, and your application makes one call instead of two. That's what's behind the 3.5x from earlier. In our testing, from a client in the same region as the index, an end-to-end search with text on an embedded field took 26 ms at the median and 39 ms at p90. Embedding first and then searching took 92 ms and 109 ms. Your numbers will depend on where your index and application run.
Writes follow a similar path in the other direction. The Documents API embeds each batch of documents, up to 96 at a time. The text and its vector go to the write-ahead log together on the same log sequence number (LSN), land in the memtable, and flush into slabs that hold the full-text index and the vector index side by side. A text clause and an embed clause therefore read the same slabs and have the same consistency guarantees.
Where this is going: a retrieval pipeline in a single API call
On just a handful of documents, either search mode tends to work fine. In a production use case your search application will likely receive both types of queries, so you’ll typically want to run both types of searches, but that's where it gets harder. Today a search scores by one method at a time, so using both means running two searches and merging the results yourself. The scores don't help with the merge: full-text scored ‘sts-61’ at 3.38 with BM25, and integrated embedding scored ‘voskhod-2’ at 0.43 with cosine similarity. The two scales aren't comparable, so you can't just sort the lists together. You end up needing to fuse them by rank, or sending the combined list to one of Pinecone’s cross-encoder rerank models.
This is the next piece we want to move inside Pinecone: one search that combines full-text and embedding clauses and fuses the results. Alongside it, we'll be adding a wider choice of popular embedding models. We’re also planning to add integrated reranking, import with integrated embedding, multimodal search, and support for data zone processing.
What's next for the Documents API: more embedding models and fusion are up next, with reranking, bulk import embedding, multimodal search, and data zone processing planned after that.
If you're starting something new, start with the Documents API. It is where full-text search, native vectors, and integrated embedding live together, and it is the shape we'll continue to build on. The integrated embedding guide walks through creating an index, writing documents, and running your first search.
If you bring your own vectors today, a document index can still hold them as native vectors and can support full-text search on the same documents. If you’re using the Records API, a document index will be more efficient while still providing the same text-in, text-out pattern, along with full-text search. We’re also working to support migrations to a document index.
Preparing your data and orchestrating the query used to be your job. With full-text search and integrated embedding in the Documents API, we’ve taken a step towards making it our job. We're not done yet, and we’re hiring. If you want to help make vectors just one of many implementation details, come build it with us.
Was this article helpful?
Integrated embedding on the Documents API
Full Text Search: Architecture and Design
Retrieval Inference for scale and performance
Silas, Adam, Rajat, Gareth
Start building knowledgeable AI today
Create your first index for free, then pay as you go when you're ready to scale.
原文为英文,正文由自动抓取并保留原文。
原文发布于 Pinecone Blog(When vectors became an implementation detail),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。