功耗与散热受限,10 万+ 加速器集群改用多数据中心

AI 集群的快速增长正推动计算基础设施突破单个数据中心的物理与功耗限制,由此催生了对新型网络架构的需求——这类架构要能够跨多个设施乃至不同地理区域连接计算资源。在 Supercomputing 2026(SC26)上,来自俄亥俄州立大学的研究人员讨论了高性能网络技术的演进。在会上,他们解释了如此规模的高效分布式计算所需的远不止更快的网络连接,并强调了跨领域扩展规模的重要性,以便把工作负载铺展到位于不同地理区域的多个数据中心。
现代 AI 集群通常可容纳超过 10 万个 AI 加速器,以及可能多达数百万个 CPU 端点,这对网络带宽、延迟和拥塞管理提出了极高要求。与此同时,加速器的计算性能提升速度持续快于通信能力,这意味着更快的 AI GPU 未必能带来应用性能的同比提升。因此,网络基础设施的效率决定了大规模 AI 系统能在多大程度上有效利用其计算资源。
传统上,高性能计算系统依赖两大互连域:scale-up(纵向扩展)和 scale-out(横向扩展)。Scale-up 网络通过 AMD 的 Infinity Fabric、Intel 的 UPI、NVIDIA 的 NVLink,或 UALink 等新兴替代方案,将紧密集成的系统内部的处理器与加速器连接起来。Scale-out 网络通常依托以太网、InfiniBand、HPE Slingshot,或更近期的 Ultra Ethernet,把单台服务器或机架级系统连接成更大的集群。这两类互连对带宽、延迟和可扩展性的要求各不相同,因此用途也不同。
然而,现代 AI 集群庞大的规模与功耗,催生了对第三个互连域的需求,即 scale-across。与通常在单个数据中心内连接系统的 scale-out 网络不同,scale-across 架构将高性能连接延伸到不同建筑、园区,甚至不同区域之间。
这种方式让运营方能够把多个位置的计算及其他资源组合起来,以达到某一水平的性能,而不是把过多的加速器集中到单个数据中心——由于供电可用性、散热需求或物理空间限制,后者可能很难,甚至根本无法实现。
向 scale-across 架构的转变已经开始。Google 今年早些时候推出了其 Virgo 网络架构,以支持超出单个数据中心空间与电力容量的 AI 工作负载。该公司打算把整个园区的计算设施当作一个单一系统来对待,尽管仍需要专门的网络来实现加速器、服务器和数据中心之间的通信。Cisco 和 Marvell 也在开发用于 scale-across 部署的技术,包括高容量路由器、交换芯片,以及旨在连接地理上分布式集群的光互连。
将 AI 基础设施扩展到单个设施之外会带来巨大的技术挑战。尽管现代 scale-up 网络能在相对较短的距离内提供极高的带宽和低延迟,但数据中心之间的通信必须应对长距离连接的物理限制。即便是光网络也无法消除传播延迟,而拥塞、丢包和同步开销还会进一步降低有效性能。因此,用高速链路连接两个大型 GPU 集群,并不一定能让它们表现得像一个紧密集成的计算系统。
这些挑战对 AI 训练尤为重要,因为加速器需要频繁交换模型参数、梯度和中间结果。大规模工作负载需要高效的集合通信,因为影响单个加速器的延迟会大幅降低整个集群的性能。因此,分布式计算的效率不仅取决于网络的原始带宽,还取决于网络拓扑、拥塞管理、通信协议和软件实现。
为了解决这些局限,网络适配器和交换机正变得前所未有地复杂。现代网络接口控制器能够在无需 CPU 持续参与的情况下执行远程直接内存访问(RDMA)操作,而 HPE Slingshot 硬件集合通信和 Nvidia SHARP 等技术则可以把某些通信操作卸载到网络基础设施自身。SmartNIC 和数据处理单元在专用硬件中处理部分网络与基础设施功能,从而进一步减轻主机处理器的负担。这类能力可以在未必需要更快的物理互连的前提下,提升分布式应用的效率。
有趣的是,软件在大规模 AI 基础设施的性能中扮演着同样重要的角色,因为 libfabric、MPI 或 UCX 等通信框架决定了应用如何交换数据、如何利用现代网络适配器的能力。此外,据研究人员称,GPU 感知通信以及 GPUDirect RDMA 之类的技术可以消除经由主机内存的不必要传输,而经过优化的集合通信操作则能降低大量加速器之间的通信开销。
AI 基础设施日益增长的复杂性也改变了运维人员评估网络性能的方式。尽管业界不断推出更快的互连技术,包括标称带宽为 800 Gb/s 和 1.6 Tb/s 的各类 GbE,但仅凭峰值传输速率并不能完整描述某一系统的能力,因为实际应用的性能取决于多种因素,包括网络硬件与软件如何有效协同来传输数据、防止拥塞,并在数千个处理器之间同步计算。
归根结底,跨规模互连架构的出现反映了 AI 基础设施设计上的根本性变化。随着单个加速器的计算能力不断提升,以及 AI 集群扩展到单个设施之外,高效通信变得与处理器性能同等重要。因此,下一代 AI 系统不仅取决于更快的 AI 加速器或 CPU,还取决于精密的网络适配器、交换机,而且或许更重要的是,取决于能够将地理上分布的计算资源整合起来、又不产生过高通信开销的软件。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Power and cooling limits force AI clusters with 100,000+ accelerators to use multiple data centers)
The rapid growth of AI clusters is pushing computing infrastructure beyond the physical and power limitations of individual data centers, which creates demand for new networking architectures capable of connecting compute resources across multiple facilities and even geographical regions. At Supercomputing 2026 (SC26), researchers from The Ohio State University discussed the evolution of high-performance networking technologies. At the conference, they explained how efficient distributed computing at such a scale requires considerably more than faster network connections, highlighting the importance of scale across domains to stretch workloads across different data centers in separate geographical regions.
Massive systems call for new interconnects
Modern AI clusters typically accommodate more than 100,000 AI accelerators and potentially millions of CPU endpoints, placing tremendous demands on network bandwidth, latency, and congestion management. Meanwhile, an accelerator's computational performance continues to increase faster than communication capabilities, which means that faster AI GPUs do not necessarily deliver proportional improvements in application performance. As a result, the efficiency of network infrastructure determines how effectively large-scale AI systems can use their computational resources.

Traditionally, high-performance computing systems have relied on two major interconnect domains: scale-up and scale-out. Scale-up networks connect processors and accelerators within tightly integrated systems using technologies like AMD's Infinity Fabric, Intel's UPI, Nvidia's NVLink, or emerging alternatives such as UALink. Scale-out networks, which typically rely on Ethernet, InfiniBand, HPE Slingshot, or, more recently, Ultra Ethernet, connect individual servers or rack-scale systems into larger clusters. These interconnects have different requirements for bandwidth, latency, and scalability, which is why they serve different purposes.
Scale-across interconnects are (almost) here
However, the massive size and power consumption of modern AI clusters create demand for a third interconnect domain, known as scale-across. Unlike scale-out networks, which typically connect systems within a single data center, scale-across architectures extend high-performance connectivity between different buildings, campuses, or even regions.
Such an approach enables operators to combine computational and other resources from multiple locations to achieve a certain level of performance rather than to bring an excessive number of accelerators to a single data center, which may be hard, or even impossible, due to power availability, cooling requirements, or physical space constraints.

The transition toward scale-across architectures has already begun. Google introduced its Virgo network architecture earlier this year to support AI workloads that exceed the space and power capacity of individual data centers. The company intends to treat an entire campus of computing facilities as a single system, albeit with specialized networks for communication between accelerators, servers, and data centers. Cisco and Marvell are also developing technologies for scale-across deployments, including high-capacity routers, switching chips, and optical interconnects designed to connect geographically distributed clusters.
Not without caveats
Extending AI infrastructure beyond individual facilities introduces substantial technical challenges. While modern scale-up networks offer extremely high bandwidth and low latency over relatively short distances, communication between data centers must contend with the physical limitations of long-distance connections. Even optical networks cannot eliminate propagation delays, whereas congestion, packet loss, and synchronization overhead can further reduce effective performance. Consequently, connecting two large GPU clusters with high-speed links does not necessarily make them behave like one tightly integrated computing system.
These challenges are particularly important for AI training, where accelerators frequently exchange model parameters, gradients, and intermediate results. Large-scale workloads require efficient collective communications, as delays that affect individual accelerators dramatically reduce the performance of the entire cluster. Therefore, the efficiency of distributed computing depends not only on raw network bandwidth, but also on network topology, congestion management, communication protocols, and software implementations.
More sophisticated hardware and software needed
To address these limitations, network adapters and switches are becoming more sophisticated than ever. Modern network interface controllers can perform Remote Direct Memory Access (RDMA) operations without continuous CPU involvement, whereas technologies such as HPE Slingshot hardware collectives and Nvidia SHARP can offload certain communication operations to the network infrastructure itself. SmartNICs and data processing units further reduce the burden on host processors as they handle selected network and infrastructure functions in dedicated hardware. Such capabilities can improve efficiency of distributed applications without necessarily requiring faster physical interconnects.
Interestingly, software plays an equally important role in the performance of large-scale AI infrastructure, as communication frameworks like libfabric, MPI, or UCX determine how applications exchange data and use the capabilities of modern network adapters. Also, GPU-aware communication and technologies like GPUDirect RDMA can eliminate unnecessary transfers through host memory, while optimized collective operations can reduce communication overhead across large numbers of accelerators, according to the researchers.
The growing complexity of AI infrastructure also changes how operators should evaluate network performance. Although the industry continues to introduce faster interconnect technologies, including versions of GbE with nominal bandwidths of 800 Gb/s and 1.6 Tb/s, peak transfer rates alone do not fully describe the capabilities of a given system because the performance of actual applications depends on multiple factors, including how effectively network hardware and software cooperate to transfer data, prevent congestion, and synchronize computations across thousands of processors.
The evolution continues
Ultimately, the emergence of scale-across interconnection architectures reflects a fundamental change in the design of AI infrastructure. As the computational capabilities of individual accelerators increase and AI clusters expand beyond individual facilities, efficient communication becomes just as important as processor performance. As a result, the next generation of AI systems will depend not only on faster AI accelerators or CPUs, but also on sophisticated network adapters, switches, and, perhaps more importantly, software that can unite geographically distributed computational resources without excessive communication overhead.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Tom's Hardware(Power and cooling limits force AI clusters with 100,000+ accelerators to use multiple data centers),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。