Power and cooling limits push 100,000+ accelerator AI clusters past single data centers

The rapid growth of AI clusters is pushing computing infrastructure beyond the physical and power limitations of individual data centers, which creates demand for new networking architectures capable of connecting compute resources across multiple facilities and even geographical regions. At Supercomputing 2026 (SC26), researchers from The Ohio State University discussed the evolution of high-performance networking technologies. At the conference, they explained how efficient distributed computing at such a scale requires considerably more than faster network connections, highlighting the importance of scale across domains to stretch workloads across different data centers in separate geographical regions.
Massive systems call for new interconnects
Modern AI clusters typically accommodate more than 100,000 AI accelerators and potentially millions of CPU endpoints, placing tremendous demands on network bandwidth, latency, and congestion management. Meanwhile, an accelerator's computational performance continues to increase faster than communication capabilities, which means that faster AI GPUs do not necessarily deliver proportional improvements in application performance. As a result, the efficiency of network infrastructure determines how effectively large-scale AI systems can use their computational resources.

Traditionally, high-performance computing systems have relied on two major interconnect domains: scale-up and scale-out. Scale-up networks connect processors and accelerators within tightly integrated systems using technologies like AMD's Infinity Fabric, Intel's UPI, Nvidia's NVLink, or emerging alternatives such as UALink. Scale-out networks, which typically rely on Ethernet, InfiniBand, HPE Slingshot, or, more recently, Ultra Ethernet, connect individual servers or rack-scale systems into larger clusters. These interconnects have different requirements for bandwidth, latency, and scalability, which is why they serve different purposes.
Scale-across interconnects are (almost) here
However, the massive size and power consumption of modern AI clusters create demand for a third interconnect domain, known as scale-across. Unlike scale-out networks, which typically connect systems within a single data center, scale-across architectures extend high-performance connectivity between different buildings, campuses, or even regions.
Such an approach enables operators to combine computational and other resources from multiple locations to achieve a certain level of performance rather than to bring an excessive number of accelerators to a single data center, which may be hard, or even impossible, due to power availability, cooling requirements, or physical space constraints.

The transition toward scale-across architectures has already begun. Google introduced its Virgo network architecture earlier this year to support AI workloads that exceed the space and power capacity of individual data centers. The company intends to treat an entire campus of computing facilities as a single system, albeit with specialized networks for communication between accelerators, servers, and data centers. Cisco and Marvell are also developing technologies for scale-across deployments, including high-capacity routers, switching chips, and optical interconnects designed to connect geographically distributed clusters.
Not without caveats
Extending AI infrastructure beyond individual facilities introduces substantial technical challenges. While modern scale-up networks offer extremely high bandwidth and low latency over relatively short distances, communication between data centers must contend with the physical limitations of long-distance connections. Even optical networks cannot eliminate propagation delays, whereas congestion, packet loss, and synchronization overhead can further reduce effective performance. Consequently, connecting two large GPU clusters with high-speed links does not necessarily make them behave like one tightly integrated computing system.
These challenges are particularly important for AI training, where accelerators frequently exchange model parameters, gradients, and intermediate results. Large-scale workloads require efficient collective communications, as delays that affect individual accelerators dramatically reduce the performance of the entire cluster. Therefore, the efficiency of distributed computing depends not only on raw network bandwidth, but also on network topology, congestion management, communication protocols, and software implementations.
More sophisticated hardware and software needed
To address these limitations, network adapters and switches are becoming more sophisticated than ever. Modern network interface controllers can perform Remote Direct Memory Access (RDMA) operations without continuous CPU involvement, whereas technologies such as HPE Slingshot hardware collectives and Nvidia SHARP can offload certain communication operations to the network infrastructure itself. SmartNICs and data processing units further reduce the burden on host processors as they handle selected network and infrastructure functions in dedicated hardware. Such capabilities can improve efficiency of distributed applications without necessarily requiring faster physical interconnects.
Interestingly, software plays an equally important role in the performance of large-scale AI infrastructure, as communication frameworks like libfabric, MPI, or UCX determine how applications exchange data and use the capabilities of modern network adapters. Also, GPU-aware communication and technologies like GPUDirect RDMA can eliminate unnecessary transfers through host memory, while optimized collective operations can reduce communication overhead across large numbers of accelerators, according to the researchers.
The growing complexity of AI infrastructure also changes how operators should evaluate network performance. Although the industry continues to introduce faster interconnect technologies, including versions of GbE with nominal bandwidths of 800 Gb/s and 1.6 Tb/s, peak transfer rates alone do not fully describe the capabilities of a given system because the performance of actual applications depends on multiple factors, including how effectively network hardware and software cooperate to transfer data, prevent congestion, and synchronize computations across thousands of processors.
The evolution continues
Ultimately, the emergence of scale-across interconnection architectures reflects a fundamental change in the design of AI infrastructure. As the computational capabilities of individual accelerators increase and AI clusters expand beyond individual facilities, efficient communication becomes just as important as processor performance. As a result, the next generation of AI systems will depend not only on faster AI accelerators or CPUs, but also on sophisticated network adapters, switches, and, perhaps more importantly, software that can unite geographically distributed computational resources without excessive communication overhead.