Google 开源 TPU Sync KV 缓存传输库
Google Cloud 发布了 TPU Sync,这是一个开源 C++ 库,用于在 Cloud TPU 上搬运 KV 缓存和模型权重。Google TPU Sync 这款 KV 缓存传输库将 TPU 高带宽内存(HBM)、主机 DRAM、共享存储和数据中心网络视为一个统一的内存层级。它瞄准的是一个日益突出的瓶颈:搬运模型状态,而非计算模型状态。Google 报告称,在 235B 至 480B 参数的混合专家(MoE)模型上,KV 传输延迟降低超过 90%。
TPU Sync 是 Google 推出的开源 C++ 库,通过异步 DMA 在多个内存层级之间搬运 TPU 分块张量。它承担两项任务:推理过程中的 KV 缓存状态,以及强化学习(RL)中从训练器到采样器的权重广播。该代码库采用 Apache-2.0 许可。
该库覆盖 TPU 高带宽内存(HBM)、锁页主机 DRAM、共享存储以及多主机 DCN。存储层使用实验性 POSIX 后端,包括本地 SSD 和 Google Cloud Managed Lustre。Google 列出了 4 项设计选择:
在此之上是一个用于前缀查找的全局 KV 存储、一个传输控制器和一个 KV 缓存管理器。
预填充(prefill)节点通过异步 DMA 将 KV 块卸载到锁页主机 RAM。控制器将这些块以主机到主机的方式流式传输到指定的解码(decode)节点。抵达的块在解码继续进行的同时移入解码节点的 HBM。
Google 于 2026 年 7 月在 TPU v7x(2x2x1)上进行了内部测试。软件栈为 tpu-inference v0.24.0、JAX 0.10.2 和 libtpu 0.0.42.1。Google 团队表示,实际结果会有所不同。
对于 P/D 分离式推理服务:
对于前缀缓存:
TPU Sync 通过 TorchTPU 与 PyTorch 集成,并原生支持 JAX。在 vLLM-TorchTPU 中,它是 vLLM 的 KVConnector 接口背后的 TPU KV 连接器。在 SGLang-TorchTPU 中,它是 P/D 传输后端和分层缓存 I/O 后端。在 GKE 上,llm-d 1.0 将其用于前缀缓存亲和性路由,并实验性地溢出至 Managed Lustre。Inferact 和 RadixArk 共同参与了 vLLM 与 SGLang 集成的工程设计。Google 表示,Meta 和 Wayfair 正在评估该架构。
这些带宽数字测量的是不同的路径。TPU Sync 的数字是主机 PCIe DMA。Mooncake 的数字是经由 RoCE 的网络传输。
需要与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等?欢迎联系我们
文章《Google 发布 TPU Sync:面向 TPU 的开源 KV 缓存传输》首发于 MarkTechPost。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Google Releases TPU Sync: Open-Source KV Cache Transfer for TPUs)
Google Cloud has released TPU Sync, an open-source C++ library for moving KV caches and model weights on Cloud TPUs. The Google TPU Sync KV cache transfer library treats TPU HBM, host DRAM, shared storage and the data center network as one memory hierarchy. It targets a growing bottleneck: moving model state, not computing it. Google reports over 90% lower KV transfer latency on 235B to 480B parameter MoE models.
TL;DR
- Library: Native C++ core with Python wheels for PyTorch and JAX. Tested on 235B to 480B MoE models.
- Runs on: Cloud TPU v7x (Ironwood) and v6e per the README. Benchmarks used TPU v7x (2x2x1), PCIe Gen5 and 200 Gbps NICs.
- Performance: Large cuts to KV transfer latency and time to first token (TTFT) in Google’s internal lab tests.
- Best: Up to 90% lower average TTFT (480B P/D serving at low concurrency, and 64K-context prefix caching).
- Worst: No RL numbers published. The README says it is not yet recommended for general use.
- Bottom line: Best: Qwen3 235B KV transfer drops from several seconds to ~230 ms. Worst: every number is self-reported and early-stage.
What is TPU Sync?
TPU Sync is an open-source C++ library from Google that moves TPU-tiled tensors across memory tiers with asynchronous DMA. It handles 2 jobs: KV cache state during inference, and trainer-to-sampler weight broadcast in reinforcement learning (RL). The repository is licensed Apache-2.0.
How does TPU Sync work?
The library spans TPU HBM, pinned host DRAM, shared storage and multi-host DCN. Storage uses experimental POSIX backends, including local SSDs and Google Cloud Managed Lustre. Google lists 4 design choices:
- Layout-preserving direct DMA: It reads JAX PJRT buffers or PyTorch tensors in place. Data keeps its native TPU tile layout, with no Python serialization or bounce buffers.
- Compute-overlapped transfers: Raw PJRT buffer aliases let transfers run during computation. Built-in DMA pacing stops large copies from delaying kernel launches.
- No XLA compilation in the data path: Transfers and resharding run in native C++ outside the XLA graph.
- High-bandwidth PCIe and DCN: It reaches up to 223 GB/s per Ironwood host, against a 256 GB/s PCIe Gen5 ceiling. DCN streaming hits ~192 Gbps on 1 NIC, or ~350 Gbps across 2.
Above this sit a global KV store for prefix lookups, a transfer controller and a KV cache manager.
How does disaggregated prefill/decode serving use it?
The prefill node offloads KV blocks to pinned host RAM through async DMA. The controller streams them host-to-host to the assigned decode node. Arriving blocks move into decode-node HBM while decoding continues.
What do the TPU Sync benchmarks show?
Google ran internal tests in July 2026 on TPU v7x (2x2x1). The stack was tpu-inference v0.24.0, JAX 0.10.2 and libtpu 0.0.42.1. Google team states that real-world results will vary.
For P/D disaggregated serving:
- KV transfer overhead fell by over 90%. Qwen3 235B median KV transfer latency dropped from several seconds to ~230 ms.
- Average TTFT fell 60% for 235B and 90% for 480B at low concurrency.
- Token output throughput rose 30% to 70% for 235B during traffic spikes.
For prefix caching:
- Effective cache capacity grew by an order of magnitude when MoE weights fill most of HBM.
- Average TTFT fell by up to 90% on 64K-context workloads.
- Throughput rose over 50%, and P99 inter-token latency fell 30%.
Which frameworks does TPU Sync support?
TPU Sync integrates with PyTorch through TorchTPU and with JAX natively. In vLLM-TorchTPU, it is the TPU KV connector behind vLLM’s KVConnector interface. In SGLang-TorchTPU, it is the P/D transfer backend and hierarchical cache I/O backend. On GKE, llm-d 1.0 uses it for prefix-cache-affinity routing and experimental spill to Managed Lustre. Inferact and RadixArk co-engineered the vLLM and SGLang integrations. Google says Meta and Wayfair are evaluating the architecture.
How does TPU Sync compare with GPU-side alternatives?
| Feature | Google TPU Sync | NVIDIA Dynamo KVBM + NIXL | LMCache | Mooncake |
|---|---|---|---|---|
| Target hardware | Cloud TPU v7x, v6e (README) | NVIDIA CUDA GPUs; partial ROCm in NIXL (NIXL) | CUDA and ROCm (repo) | CUDA, HIP, Ascend, MUSA and more (repo) |
| Memory tiers | HBM, pinned host DRAM, POSIX storage (experimental), DCN | GPU, CPU, local SSD, remote storage (KVBM docs) | GPU, CPU DRAM, local disk, remote backends | DRAM and SSD/NVMe (Mooncake Store) |
| Network transport | DCN, multi-NIC striping | UCX, LIBFABRIC, GDS plugins | NVLink, RDMA, TCP | TCP, RDMA, EFA, NVMe-oF, NVLink, CXL |
| Engine integrations | vLLM-TorchTPU, SGLang-TorchTPU, llm-d, JAX | vLLM, TensorRT-LLM, SGLang (KVBM) | vLLM, NVIDIA Dynamo | vLLM, SGLang |
| License | Apache-2.0 | Apache 2.0 (some wheel modules NVIDIA proprietary) | Apache 2.0 | Apache-2.0 |
| Core language | C++ with Python bindings | C++ with Python and Rust bindings | Python package with C/C++ and Rust code | Not disclosed |
| Headline bandwidth | Up to 223 GB/s per v7x host (PCIe D2H/H2D) | Not disclosed | Not disclosed | Up to 190 GB/s on 8×400 Gbps RoCE (network) |
| RL weight transfer | Yes (experimental) | Not disclosed | Not disclosed | Yes, via SGLang (RDMA) |
The bandwidth figures measure different paths. TPU Sync’s number is host PCIe DMA. Mooncake’s number is network transfer over RoCE.
Key Takeaways
- TPU Sync is Google’s Apache-2.0 C++ library for KV cache and weight movement on TPUs.
- It reaches up to 223 GB/s per Ironwood host through direct, layout-preserving DMA.
- Qwen3 235B KV transfer fell from several seconds to ~230 ms in Google’s tests.
- It plugs into vLLM-TorchTPU, SGLang-TorchTPU and llm-d on GKE.
- RL and storage tiering are experimental, and every number is self-reported.
Check out the technical blog, the GitHub repo, vLLM-TorchTPU and SGLang-TorchTPU. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Google Releases TPU Sync: Open-Source KV Cache Transfer for TPUs appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Google Releases TPU Sync: Open-Source KV Cache Transfer for TPUs),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。