每字节榨出更多性能:全栈协同设计破解 AI 内存挑战

更大的模型、更长的上下文以及智能体式应用正在加大 GPU 内存压力,与此同时供应趋紧、价格上涨——这一趋势预计在短期内仍将持续。
NVIDIA 与内存合作伙伴紧密协作,快速采用最新的内存技术。NVIDIA 率先将 HBM3e 和 HBM4 技术引入加速计算技术栈。
但这一做法并非简单地在 GPU 上塞进更多内存,而是全栈协同设计。NVIDIA 有意将 GPU、CPU、网络、存储和软件放在一起设计,以便在 AI 工厂的每一层从内存中挖掘出更多有效工作。
这一点很重要,因为 AI 工厂运行在内存与存储的分层体系之上。GPU 高带宽内存(HBM)是最快、最昂贵、争夺最激烈的一层——模型权重和 KV 缓存争抢着有限的容量。其下是 CPU 内存、本地 SSD 和共享存储,每一层都以速度换取更大的容量和更低的单位比特成本。
一个字节能完成多少有效工作,取决于整个分层体系被用得有多好:压缩必须存储的内容、把它放到合适的层级,并在需要之前把它送回 GPU。
逐字节来看,NVIDIA 在每一层都实现了最高的内存使用效率——结果也印证了这一点。从 NVIDIA Blackwell Ultra 到 Vera Rubin 平台,每 GPU 的 HBM 容量提升了约 3%,而在 SemiAnalysis AgentX 上,每美元的智能体吞吐提升最高达 30 倍。
前沿模型如今拥有数万亿参数——这些数值承载着 AI 模型在训练中学到的东西。
KV 缓存用于保存模型理解每个活跃请求所需的上下文,从而避免重复处理相同信息,而它也在不断膨胀。智能体式轨迹会在反复的模型调用和工具调用中携带先前的上下文,某些情况下处理的 Token 数高达 300 万,因此每个请求的工作记忆会随时间不断增长。
检索增强生成(RAG)将相关的企业信息加入提示词,从而产生更多需要处理和保留的上下文。
在推理过程中,高带宽内存(HBM)主要承载模型参数和 KV 缓存。结果就是:HBM 承受的压力更大。
为每块 GPU 增加更多 HBM 容量是应对这种压力的直接办法,但容量能发挥多少,取决于带宽能搬运多少。
可以把 HBM 想象成一座仓库,内存带宽则是它的装卸码头。仓库越大,能存放的库存越多,但码头决定了同一时间能进出多少货物。一旦码头满负荷,搬运更多库存就需要更长时间,响应速度随之放缓。如果延迟超出所需的服务水平,多出来的空间就无法有效利用,反而会增加成本、消耗电力。
这也正是为什么没有哪家仓储运营商会把所有库存都放在最近、最快的空间里。正在流转的货物留在产线附近,等待中的货物则转移到附近更大、更便宜的仓位,并在需要之前返回。
AI 工厂面临同样的限制。
更多 HBM 可以容纳更多 KV 缓存,支持更多并发用户。每增加一个用户,都要共享同样的算力和内存带宽。但当每个用户都期待快速响应时,内存带宽最终无法以算力消耗数据的速度供应数据:工程师将这一限制称为内存墙。
超过这个限度,额外的 HBM 容量便无法发挥作用,却仍会推高成本、消耗电力。数据依然需要有存放之处,但并非所有数据都需要最快、最昂贵的层级。这正是 AI 工厂要在内存与存储的层级体系中管理数据的原因。
NVIDIA 的全栈协同设计通过缓解模型权重、KV 缓存和 RAG 数据带来的压力,让每一字节 HBM 都物尽其用。它采用的技术各自针对不同的效率瓶颈来源:
压缩模型权重,并将其分布到整个 NVL72 域中。
压缩 KV 缓存,并将不活跃的上下文卸载到成本更低的存储层级。
压缩 RAG 嵌入,并将索引保存在 HBM 之外。
结果是,用于模型权重的 HBM 更少,被闲置上下文占用的 HBM 更少,而有更多 HBM 可用于服务并发用户。
在 NVIDIA 平台的历次迭代中,每美元每秒 Token 数的提升速度远超 HBM 容量的增长。
在 SemiAnalysis AgentX DeepSeek V4 Pro 工作负载上,在每位用户每秒 100 Token 的条件下,NVIDIA GB300 NVL72 的每美元吞吐最高可达 NVIDIA H200 平台的 10 倍,而每 GPU 的 HBM 容量约为其两倍。在每位用户每秒 125 Token 的条件下,Vera Rubin NVL72 的每美元吞吐最高可达 GB300 NVL72 的 30 倍,而每 GPU 的 HBM 容量仅增加约 3%。
对 AI 工厂运营者而言,真正重要的指标是:在满足用户响应时间需求的前提下,工厂能在其功耗预算内输出多少有用的 Token。每瓦每秒 Token 数越高,意味着同样的功耗能完成更多工作、带来更多收入,同时每个 Token 的成本更低。NVIDIA 的全栈协同设计通过提升每一字节内存所交付的有效工作量,降低了 Token 成本。
想了解更多,可观看视频讲解与技术深度解析,了解 NVIDIA 全栈协同设计如何缓解推理内存压力。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge)
Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise — a trend expected to continue through the near term.
NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack.
But the approach isn’t to simply pack more memory onto the GPU. It’s full-stack codesign. NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory.
That matters because an AI factory runs on a memory and storage hierarchy. GPU high-bandwidth memory (HBM) is the fastest, most expensive and most contested tier — with model weights and KV cache competing for limited capacity. Below it sit CPU memory, local SSDs and shared storage, each trading speed for more capacity at a lower cost per bit.
How much useful work a byte does depends on how well the whole hierarchy is used: shrinking what must be stored, placing it on the right tier and bringing it back to the GPU before it’s needed.
Byte for byte, NVIDIA makes the most efficient use of memory at every layer — and the results show it. From NVIDIA Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3%, while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.
Agentic Inference Is Putting More Pressure on HBM
Frontier models now have trillions of parameters — the values that hold what an AI model learned in training.
KV cache, which holds the context the model needs to understand each active request without reprocessing the same information, is also expanding. Agentic trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens, so each request’s working memory grows over time.
Retrieval-augmented generation (RAG) adds relevant enterprise information to prompts, creating even more context to process and retain.
During inference, HBM primarily holds both model parameters and KV cache. The result: greater pressure on HBM.

Why AI Factories Need a Memory Hierarchy
Adding more HBM capacity per GPU is the obvious response to that pressure, but capacity delivers only what bandwidth can move.
Think of HBM as a warehouse and memory bandwidth as its loading docks. A larger warehouse holds more inventory, but the docks set how much can move in and out at a time. Once the docks are at capacity, moving more inventory takes longer and slows response times. If delays exceed the required service level, the added space can’t be used effectively, even as it adds cost and draws power.
That’s also why no operation keeps all its inventory in the closest, fastest space. What’s in motion stays near the line. What’s waiting moves to a larger, cheaper space nearby and returns before it’s needed.
An AI factory faces the same limit.
More HBM can hold more KV cache and support more concurrent users. Every added user shares the same compute and memory bandwidth. But when each user expects fast responses, eventually memory bandwidth is unable to supply data as fast as compute can use it: a limit engineers call the memory wall.
Past that point, extra HBM capacity can’t be put to work, yet it still adds costs and draws power. The data still needs a place to live, but not all of it needs the fastest, most expensive tier. This is why an AI factory manages data across a memory and storage hierarchy.

Maximizing the Value of Every Byte With Full-Stack Codesign
NVIDIA’s full-stack codesign makes every byte of HBM count by relieving pressure from model weights, KV cache and RAG data. It does this with techniques that each target a different source of inefficiency:

Shrink model weights and distribute them across the NVL72 domain.
- The NVFP4 format reduces the space required for weights compared to FP8, the lowest precision possible on the NVIDIA Hopper platform. Recent models, including FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark from Meta, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL have been released with NVFP4 checkpoints, demonstrating growing model-builder adoption of 4-bit precision for faster, more memory-efficient inference.
- Within an NVL72 system, model weights can be distributed across up to 72 GPUs, so each GPU holds a smaller share of the model — leaving more HBM for KV cache and concurrent users.
Shrink KV-cache and offload inactive context to lower-cost storage tiers.
- Applying NVFP4 to the KV cache reduces its footprint compared with FP8, so more context can stay resident in HBM.
- NVIDIA Vera CPU helps in two ways. Faster tool execution shortens the time KV cache sits inactive during tool calls, and its large LPDDR memory, connected to NVIDIA Rubin GPUs over NVIDIA NVLink-C2C interconnect, gives paused context a nearby place to wait and return quickly.
- When context isn’t actively in use, NVIDIA Dynamo inference software moves it to NVIDIA CMX context memory storage — a tier built for capacity at far lower cost per bit than HBM — and returns it before reuse. Unlike local SSDs, CMX is shared across servers, and unlike enterprise storage, it skips the overhead of long-term data protection that short-lived context doesn’t need. NVIDIA CMX context memory storage partners include Cloudian, DDN, Dell Technologies, Everpure (previously Pure Storage), HPE, IBM, NetApp, VAST and WEKA.
- Dynamo’s KV Cache Reuse ties the tiers together, managing KV cache as one system-wide resource across GPU memory, CPU memory and storage, so context computed once can be reused across the system.
Compact RAG embeddings and keeping the index outside HBM.
- RAG turns enterprise documents into embeddings — numerical representations that let a system search by meaning rather than keyword — organized into a searchable index. NVIDIA Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy, while the NVIDIA cuVS library enables that index to be built and served across CPU and storage. The knowledge base never occupies HBM, and only the passages the model needs enter context.
The result is less HBM used for model weights, less HBM tied up by idle context and more of it available to serve concurrent users.
More Useful Output per Dollar With Each Generation
Across successive NVIDIA platform generations, tokens per second per dollar has risen far faster than HBM capacity.
On the SemiAnalysis AgentX DeepSeek V4 Pro workload, NVIDIA GB300 NVL72 delivers up to 10x higher throughput per dollar than NVIDIA H200 platform at 100 tokens per second per user, with HBM capacity per GPU roughly doubling. At 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, with HBM capacity per GPU up about 3%.

For AI factory operators, the metric that matters is how many useful tokens the factory can deliver within its power budget while meeting user response-time needs. More tokens per second per watt means more work and revenue from the same power, and a lower cost for every token. NVIDIA’s full-stack codesign lowers token costs by increasing the useful work every byte of memory delivers.
Learn more in the video explainer and technical deep dive on how NVIDIA full-stack codesign reduces inference memory pressure.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 NVIDIA Blog(More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。