In the first part of this book, we completed an in-depth exploration of the core "building blocks" needed to construct a large-model computing center. We understood the microarchitecture of GPUs, dissected the system design of GPU servers, and mastered the Magnum IO software I/O framework that ties it all together. We now possess powerful, optimized individual compute nodes.
Beginning with this chapter, we enter the second part of the book, raising our perspective from a single node to the macro level of the entire cluster. Our task is to combine hundreds or even thousands of these "super blocks" efficiently, stably, and reliably into a true AI supercomputer. This process is like moving from building individual villas to planning and constructing a modern metropolis with expressways, water and power networks, commercial districts, and residential areas.
The network is, without doubt, the most important infrastructure in this "AI city." It is like the city's traffic arteries, carrying various kinds of "traffic"—data, gradients, model parameters, control signaling—shuttling between different functional zones. The quality of the network design directly determines the scalability, stability, and overall performance of the entire cluster. A poorly designed network will cripple an expensive GPU cluster through communication bottlenecks, like pouring countless Ferraris into a city with no traffic lights and nothing but dead ends, the result being endless congestion and chaos.
This chapter—GPU Cluster Network Design and Implementation—serves as the opening of the second part, focusing on building this crucial "transportation network." From a network architect's perspective, we will systematically explore how to design and implement a multi-plane, high-availability, bottleneck-free network architecture for a large-scale GPU cluster.
We will follow the principle of "divide and conquer," partitioning the complex cluster network into four network planes that are logically independent and may be physically fused:
- Compute Fabric: This is the "expressway" built for gradient synchronization and model parameter exchange among GPUs during distributed training, demanding extreme low latency, high bandwidth, and lossless transmission. We will take RoCE (RDMA over Converged Ethernet) as our example and elaborate on its design and implementation in detail.
- Storage Fabric: Responsible for connecting compute nodes to the backend distributed storage system, providing a continuous, high-bandwidth data flow for loading massive training datasets.
- Business/Application Network: Provides general-purpose, reliable network connectivity for applications other than AI training (such as API services, user access, and data analysis).
- Out-of-Band Management/Monitoring Network: This is the cluster's "life support system," independent of all data networks, responsible for the remote management, monitoring, and troubleshooting of all hardware devices.
In addition, we will discuss the design of the network boundary—that is, how the cluster connects securely and efficiently to the outside world (such as the public internet and the enterprise intranet).
By the end of this chapter, you will have mastered a complete methodology for GPU cluster network design, understood the necessity of a multi-plane network architecture, and been able to delve into the specific technology choices for each plane (such as the Clos architecture, ECMP, and PFC/ECN) and their implementation details. This will lay a solid network foundation for building a stable, efficient, and scalable large-model computing platform.
6.1 Design and Implementation of a RoCE Compute Network in a GPU Cluster
The compute network is the plane with the highest performance requirements and the most critical design in a GPU cluster. Its sole mission is to provide a communication foundation with ultra-low latency, ultra-high bandwidth, and no blocking for the dense inter-GPU communication in large-scale distributed training. As discussed in Chapter 5, the technical core of achieving this goal is RDMA, and RoCE (RDMA over Converged Ethernet), thanks to its compatibility with the Ethernet ecosystem, has become a popular choice for many enterprises building AI clusters.
However, implementing high-performance, large-scale RDMA over Ethernet is an extremely challenging systems-engineering undertaking. This section will elaborate on the design principles, core technologies, and implementation steps needed to build a successful RoCE compute network.
6.1.1 Design Principles: Born for Losslessness and Low Latency
The design of a RoCE compute network must follow several core principles:
- Lossless: This is the foremost principle. The packet-loss behavior of traditional Ethernet is fatal to RDMA. We must transform the underlying network into a "lossless network" through technical means, ensuring that data packets are never dropped by switches due to congestion.
- Low Latency: The performance of distributed training is extremely sensitive to communication latency. Every aspect of the network design—from topology and switch selection to cabling—must aim to minimize end-to-end latency.
- High Bandwidth: The network must provide non-blocking bandwidth matched to the port rates of the GPU node NICs (such as 200 Gbps or 400 Gbps).
- Scalability: The network architecture must support the smooth expansion of cluster scale in the future, growing from a few dozen nodes to over a thousand without requiring disruptive changes to the core architecture.
- High Availability: No single point of failure in the network (such as a switch or link) should bring down the entire compute network.
6.1.2 Core Technology One: Clos Network Architecture
To satisfy the requirements for high bandwidth, scalability, and non-blocking performance, modern data center networks universally adopt the Clos architecture (also known as the Spine-Leaf architecture or the Fat-Tree architecture).
Architecture Components: The Clos architecture comprises two layers of switches:
Leaf Switches: Positioned at the bottom of the network, they directly connect to server nodes (GPU servers). Each server is typically connected redundantly to two different leaf switches.
Spine Switches: Positioned at the top of the network, they connect to no servers. Their sole purpose is to connect all leaf switches. In a strict Clos architecture, every leaf switch must connect to every spine switch.
Data Path:
Communication between servers always traverses exactly "one up, one down"—two hops: Server A -> Leaf A -> Spine -> Leaf B -> Server B. The path length between any two servers is fixed (three hops: server -> leaf -> spine -> leaf -> server), which guarantees predictable communication latency.
Leaf switches do not connect directly to one another; all traffic crossing between leaf switches must pass through a spine switch.
Advantages of the Clos Architecture:
Non-Blocking Bandwidth: By carefully calculating and configuring the bandwidth ratio (oversubscription ratio) between the uplinks (Leaf-Spine) and the downlinks (Leaf-Server), a theoretically non-blocking network can be achieved. For example, if a leaf switch has 24 downlink ports connecting to servers and 24 uplink ports connecting to spine switches, all at the same port rate, then its oversubscription ratio is 1:1—that is, non-blocking.
Load Balancing and Multi-Pathing: From one leaf switch to another, there exist multiple equal-cost paths through different spine switches. The network can leverage Equal-Cost Multi-Path (ECMP) routing to hash traffic across these paths, achieving load balancing and bandwidth aggregation.
High Scalability:
To increase server capacity, simply add new leaf switches and connect them to all spine switches.
As the number of leaf switches grows to the point where the spine switches run out of ports, additional spine switches can be added.
For ultra-large-scale clusters, a three-tier Clos architecture (Leaf-Spine-SuperSpine) can even be built to further enhance scalability.
High Availability: If any spine or leaf switch fails, traffic is automatically switched to other healthy paths via ECMP, without causing service interruption.
6.1.3 Core Technology Two: Implementing Lossless Ethernet
On top of the skeleton that is the Clos architecture, we need to fill in the "flesh"—the technical details for achieving lossless transmission. This relies primarily on the coordinated work of PFC and ECN.
PFC (Priority-based Flow Control, IEEE 802.1Qbb)
Working Principle: PFC refines the traditional Ethernet PAUSE frame mechanism (which pauses all traffic on a link). It allows traffic to be divided into up to 8 different priority classes (Priority Class, PC), with flow control applied independently to each class.
Application in RoCE: We typically assign a dedicated, high-priority PC (for example, PC 3) to RoCE traffic. When the buffer for PC 3 on a switch's egress port is about to fill, the switch sends a PFC PAUSE frame for PC 3 to its upstream device (another switch or a server NIC). Upon receiving it, the upstream device pauses the transmission of all traffic belonging to PC 3, while traffic of other priorities (such as management traffic and TCP traffic) remains unaffected.
Role: PFC is the first line of defense against RoCE packet drops caused by momentary congestion.
ECN (Explicit Congestion Notification, RFC 3168)
The Risk of PFC: Although PFC prevents packet loss, it can also introduce new problems. If congestion persists, PFC PAUSE frames propagate continuously upstream, possibly causing large-scale network "pauses" and even PFC storms or deadlocks. PFC only solves the problem of "plugging" the flow; it does not solve the problem of "draining" it.
How ECN Works: ECN provides a proactive congestion management mechanism. Each switch is configured with a buffer threshold (the ECN marking threshold). When the buffer occupancy of a queue exceeds this threshold, the switch does not immediately pause the traffic; instead, it sets a "Congestion Experienced" (CE) mark in the IP headers of the packets passing through that queue.
When the packet reaches the destination NIC, the NIC detects the CE mark and feeds the congestion signal back to the original sender via a special Congestion Notification Packet (CNP).
Upon receiving the CNP, the sender proactively and temporarily lowers its sending rate, thereby relieving network congestion.
DCQCN (Data Center Quantized Congestion Notification): This is the advanced congestion control algorithm widely used in RoCE networks today, combining ECN and PFC. It precisely defines how the sending rate should be adjusted in response to received CNPs, and how to coordinate with PFC.
The Coordination of PFC and ECN
ECN is a proactive, end-to-end congestion control mechanism. It attempts to "drain" traffic by slowing down early in the onset of congestion, and is the primary means of resolving congestion.
PFC is a reactive, hop-by-hop congestion avoidance mechanism. It is the last line of insurance when ECN cannot react in time or congestion is extremely severe, "plugging" the flow through pausing to prevent packet loss. In a healthy RoCE network, ECN should be doing the work the vast majority of the time, and the triggering of PFC should be a low-probability event.
6.1.4 Implementation Steps for a RoCE Compute Network
Physical Design
- Topology: Design a Spine-Leaf Clos architecture that meets future expansion needs, with the lowest possible oversubscription ratio (ideally 1:1).
- Device Selection: Choose data center switches that support PFC and ECN, have large and dynamic buffers, offer low latency, and provide high port rates (200G/400G)—for example, the NVIDIA Spectrum series, Arista, and Cisco Nexus. Choose high-performance RoCE NICs that pair with the GPU servers (such as the NVIDIA ConnectX series).
- Cabling: Use high-quality optical fibers and transceivers, precisely calculate lengths, and ensure signal quality. All cabling should be neat, orderly, and clearly labeled.
Logical Configuration
- Switch Configuration: This is the most complex part.
- Enable PFC and assign a dedicated, lossless priority to RoCE traffic (typically identified by DSCP values).
- Enable ECN and configure appropriate ECN marking thresholds for the lossless queue.
- Configure DCQCN or another congestion control algorithm.
- Configure basic network parameters such as VLANs, IP addresses, and BGP (typically used for route advertisement between Spine and Leaf).
- Configure ECMP to ensure that traffic is hashed evenly across all available paths.
- Server Configuration:
- Install and configure the NIC drivers and RDMA-related software packages in the operating system.
- Configure the server NIC so that the RoCE traffic it emits carries the correct DSCP values, enabling switches to identify it and place it into the lossless queue.
- Configure server-side congestion control so that it can respond to ECN signals.
Testing and Validation
Basic Connectivity Testing: Use tools such as ping, ib_write_bw, and ib_write_lat to test basic connectivity, bandwidth, and latency between nodes.
Congestion Testing: Use specialized tools (such as nd_stress) to simulate a variety of congestion scenarios (for example, Incast congestion), verifying that PFC and ECN behave as expected and that the network is truly lossless.
Application-Level Testing: Run small-scale All-Reduce benchmarks (such as nccl-tests) to check network performance under real application workloads.
Monitoring and Tuning
Deploy a network monitoring system to continuously collect the relevant metrics from switches and NICs, especially PFC PAUSE frame counts, ECN marking counts, and buffer occupancy.
By analyzing these metrics, one can determine whether the network harbors potential congestion points or configuration problems, and carry out continuous tuning.
Building a successful RoCE compute network requires close cooperation among the network, systems, and AI application teams. It is a closed-loop engineering effort that integrates design, implementation, testing, and monitoring.
6.2 Design and Implementation of Storage and Business Networks in a GPU Cluster
Although the compute network is the jewel in the crown of performance, the storage network and the business network are just as indispensable to keeping the cluster healthy and operating efficiently. Separating them from the compute network—logically and even physically—is an excellent design practice.
6.2.1 Why Separate the Network Planes?
Vastly Different Requirements
Compute Network: Pursues the extremes of low latency and lossless transmission; its traffic pattern consists mainly of large, bursty RDMA traffic between GPUs.
Storage Network: Pursues high throughput and stability; its traffic pattern consists mainly of continuous, large-block reads and writes between compute nodes and storage nodes.
Business Network: Pursues generality and reliability; its traffic pattern is complex and varied, containing many small packets and TCP connections.
Avoiding Interference
If all traffic were mixed into a single network, a "bad money drives out good" phenomenon could occur. For example, a sudden storm of TCP traffic could seize switch buffers, affecting the latency-ultra-sensitive RoCE traffic and causing jitter in distributed training performance.
Simplifying Management and Security
Separate network planes allow the configuration and optimization of each network to proceed independently. For example, we only need to configure the complex lossless Ethernet technologies in the compute network, while the storage and business networks can use simpler, standard Ethernet configurations. At the same time, this makes it easier to achieve security isolation between different network planes through firewalls and access control lists (ACLs).
6.2.2 Design and Implementation of the Storage Network
The goal of the storage network is to provide GPU nodes with a high-speed, reliable connection to the backend distributed storage system.
Traffic Characteristics: Primarily large-block, continuous read and write traffic between compute nodes (clients) and storage nodes (servers). Bandwidth requirements are high, but latency sensitivity is lower than in the compute network.
Technology Choices:
- Network Protocol: The storage network typically also runs over Ethernet. Depending on the backend storage system, the upper-layer protocol may be standard TCP/IP (for connecting to NFS or S3-based object storage) or RDMA (for connecting to high-performance storage that supports NVMe-oF over RoCE/InfiniBand).
- Topology Architecture: The Spine-Leaf Clos architecture can be adopted here as well. In a typical design, compute nodes serve as servers under one set of leaf switches, while the storage nodes of the distributed storage system serve as servers under another set of leaf switches, all attached to the same Spine-Leaf network.
- Bandwidth and Oversubscription Ratio: The bandwidth design of the storage network requires careful planning. It is necessary to estimate the total storage I/O bandwidth demand that all compute nodes might generate when loading data at full load, and to ensure that the aggregate bandwidth of the network can satisfy this demand. The oversubscription ratio of the storage network can be somewhat higher than that of the compute network (for example, 1:3 or 1:4), because it is usually not the case that all compute nodes reach peak storage I/O at the same moment.
Implementation Key Points:
- Jumbo Frames: Enabling jumbo frames with an MTU of 9000 bytes on the storage network can markedly improve the efficiency of large-block data transfer. For transmitting the same 1 MB of data, the number of packets required and the header overhead are far smaller than with the standard 1500-byte MTU.
- Link Aggregation (LAG/LACP): Multiple physical links from a compute or storage node to a leaf switch can be bundled into a single logical link to increase bandwidth and provide link redundancy.
- Traffic Shaping and QoS: If the storage network and the business network physically share infrastructure (for example, using VLANs for logical isolation), QoS policies must be configured to ensure that high-priority storage traffic is not affected by low-priority business traffic.
6.2.3 Design and Implementation of the Business Network
The business network is a general-purpose data plane that carries all business traffic other than compute and storage.
Traffic Types:
- Model Serving (Inference): If the cluster is also used to provide online inference services, the API requests from users and the responses of the models travel over the business network.
- User Access: When users log into compute nodes via SSH for development and debugging, that traffic goes through the business network.
- Data Analysis and Visualization: Web access traffic for services such as Jupyter Notebook and TensorBoard.
- Inter-Application Communication: Communication among the various microservices in the cluster.
- Connecting to External Data Sources: Downloading data, code, Docker images, and so on from the internet or the enterprise intranet.
Design Principles: The primary principles of business network design are reliability, generality, and security.
Topology: The Spine-Leaf architecture can also be adopted here. Sharing physical infrastructure with the storage network is a common and cost-saving practice.
Bandwidth: The bandwidth requirements of the business network are typically far lower than those of the compute and storage networks. Interfaces of 10 Gbps or 25 Gbps are usually sufficient.
Reliability: High availability must be considered; all network devices (switches, links) should have redundancy.
Security: The business network is the primary channel connecting the cluster to the outside world, and is therefore the focus of security protection. Security policies such as firewalls and ACLs need to be deployed on this network.
6.3 Design and Implementation of the Out-of-Band Management and Monitoring Network in a GPU Cluster
The out-of-band (OOB) network is the cluster's "lifeline," and its importance cannot be overemphasized. It is a dedicated management channel completely independent of all data networks (compute, storage, and business).
6.3.1 The Critical Importance of Independence
Imagine that a compute node's business NIC driver crashes, or a switch configuration error paralyzes the entire data network. How would you log into that server to troubleshoot the problem? If the management channel also depended on that paralyzed data network, you would be completely "out of reach" of that server, and your only option would be to send someone to the machine room to plug in a monitor and keyboard.
The value of the OOB network is precisely this: no matter how severe the fault in the data networks, as long as the server is powered on and plugged into the OOB cable, the administrator always has a reliable path to the device's "brain"—the BMC (Baseboard Management Controller).
6.3.2 Design and Implementation of the OOB Network
Topology and Equipment:
- The OOB network is typically a very simple, low-cost star or tree topology.
- Dedicated, inexpensive 1GbE or 10GbE management switches are used. For high availability, a redundant pair of management switches is usually deployed.
- Every device in the cluster (including the BMC ports of the GPU servers, the management ports of all compute/storage/business switches, PDUs, storage controllers, and so on) must connect to the OOB network.
Functions:
The OOB network carries all the management and monitoring traffic of the infrastructure:
- Remote Management:
- Performing power on/off, firmware upgrades, hardware configuration, and other operations on servers via the IPMI or Redfish protocols.
- Remote desktop access via KVM over IP.
- Automated deployment: Using PXE (Preboot Execution Environment) over the OOB network for unattended operating system installation.
- Monitoring Data Collection:
- Monitoring systems such as Prometheus and Zabbix collect hardware status information (temperature, power consumption, fan speed, hardware fault logs, and so on) from server BMCs, switches, and other devices via the OOB network.
- This is the foundation for achieving global visibility into cluster health.
- Configuration Management: Automated operations tools such as Ansible and SaltStack push configuration changes to all devices via the OOB network.
- Security:
- The OOB network holds the highest level of control over all infrastructure, so it must be the network with the highest security level and the strictest isolation in the entire data center.
- It must be physically isolated from all data networks.
- Access to the OOB network must be strictly controlled, typically requiring traversal through a secure bastion host (jump host).
A well-designed, stable, and reliable OOB network is the prerequisite and foundation for automated and intelligent operations (AIOps) of large-scale clusters.
6.4 Design and Implementation of the Network Boundary in a GPU Cluster
The network boundary is the gateway connecting the GPU cluster to the outside world. How this boundary is designed determines the cluster's security, accessibility, and ability to integrate with external services.
6.4.1 Functions and Challenges of the Boundary
The network boundary must carry several types of inbound and outbound traffic:
Egress Traffic:
- Applications inside the cluster need to reach the internet to download software packages, Docker images, pre-trained models, and so on.
- Training tasks may need to push logs, metrics, and other data to external monitoring or storage systems.
Ingress Traffic:
- Users and administrators need to access the cluster from the enterprise intranet or the public internet.
- If the cluster provides API services, external users need to be able to access those services.
Challenges:
- Security: How can the cluster be protected from external attacks while still providing the necessary connectivity?
- Performance: How can enough bandwidth be provided for legitimate inbound and outbound traffic so that it does not become a bottleneck?
- Management: How can traffic entering and leaving the cluster be audited and controlled?
6.4.2 A Typical Boundary Design Architecture
A common boundary design architecture adopts a multi-layered, defense-in-depth model.
Edge Routers
These are the first hop connecting the cluster to external networks (such as carrier networks and the enterprise backbone).
A redundant pair of high-performance edge routers is typically deployed, exchanging routing information with the outside world via the BGP protocol.
Firewalls
Located behind the edge routers, they are the core of security protection.
All traffic entering and leaving the cluster must pass through firewall policy inspection. Next-generation firewalls (NGFW) can be deployed to provide advanced capabilities such as stateful inspection, intrusion prevention (IPS), and application identification.
For performance reasons, a high-throughput firewall cluster is typically deployed.
Load Balancers
For applications that need to provide services externally (such as API inference services), load balancers are indispensable.
They receive requests from outside and, based on certain policies (such as round-robin or least-connections), distribute those requests across multiple backend servers, enabling horizontal scaling and high availability of the service.
Either hardware load balancers (such as F5 or A10) or software load balancers (such as HAProxy or Nginx) can be deployed.
DMZ (Demilitarized Zone)
To further enhance security, servers that need to be directly exposed to external access (such as web servers, API gateways, and bastion hosts) can be placed in a special network area called the DMZ.
The DMZ is strictly isolated by firewalls from both the cluster's internal core network (such as the compute and storage networks) and the external internet. Even if a server in the DMZ is compromised, it is difficult for an attacker to penetrate the core network directly.
NAT Gateway (Network Address Translation Gateway)
Servers inside the cluster typically use private IP addresses. When they need to access the internet, their traffic must pass through a NAT gateway, which translates their private IP addresses into one or more public IP addresses.
The NAT gateway can centrally manage and audit the cluster's egress traffic.
6.4.3 Connecting the Network Planes to the Boundary
The business network is the primary network connected to the boundary. Users' SSH access, the applications' API services, and so on all communicate with the outside world through the business network, passing through firewalls and load balancers.
The storage network and the compute network should generally not be exposed directly to the outside; they should be strictly isolated within the cluster.
The OOB management network must never be directly connected to the external internet. Remote access to the OOB network must be made through a bastion host located in the DMZ that has undergone multiple layers of security hardening.
6.5 Chapter Summary
In this chapter, we completed a crucial step from a single GPU server to building a complete GPU cluster—network design and implementation. We came to understand deeply that the network is the "nervous system" connecting thousands of computing, storage, and management units, and that the success or failure of its design directly bears on the performance, stability, and scalability of the entire AI cluster.
We adopted the core design concept of a multi-plane network, decomposing the complex cluster network into four logical planes with clear objectives and independent functions, and explored in depth the design essence of each plane:
- In the compute network, taking RoCE as our example, we systematically learned how to build a lossless, low-latency network built for RDMA. We mastered its two core technologies: the Clos (Spine-Leaf) architecture provides a non-blocking, scalable physical skeleton, while the coordinated work of PFC and ECN is the key "magic" for achieving lossless transmission over Ethernet—the former being the last line of insurance, the latter the proactive draining of congestion.
- In the design of the storage and business networks, we emphasized the importance of separating them from the compute network to avoid performance interference and simplify management. We understood the storage network's pursuit of high throughput, and the business network's emphasis on generality, reliability, and security.
- In the out-of-band management and monitoring network, we recognized that its complete physical independence is the lifeline for achieving high-availability operations and rapid fault response in the cluster. It is the "last line of insurance" connecting the BMCs of all devices, and the foundation for automated deployment, monitoring, and configuration management.
- Finally, in the design of the network boundary, we learned how to build a defense-in-depth security system through components such as firewalls, load balancers, and DMZ—maximizing resistance to security risks while ensuring the necessary connections between the cluster and the outside world.
Through the study of this chapter, we no longer view the cluster network as a mere pile of switches and cables, but can carry out global planning, layered design, and technical trade-offs from the perspective of a systems architect. We have acquired an effective methodology: whether choosing InfiniBand or RoCE, whether planning a two-tier Clos or a three-tier Clos, we can design the most suitable network solution for a GPU cluster of a given scale and requirements, weighing dimensions such as performance, cost, scalability, and maintainability.
This solid network foundation will provide a stable, efficient operating platform for the higher-level cluster resource management and virtualization technologies that we will explore next in the second part.