祝融说。

In the previous chapter, we delved into the internal architecture of the two major AI compute engines: NVIDIA and Huawei Ascend. However, a harsh reality remains: even if you possessed the world's most powerful GPU cluster, if you cannot efficiently "feed" it data and allow seamless "conversation" between the cards, these expensive silicon slabs are just high-wattage space heaters. Large model training, especially distributed training involving thousands of cards, is no longer a competition of single-point compute power, but rather a manifestation of the comprehensive capability of the entire system engineering -- especially networking and storage.

Imagine a massive F1 team. The race car engine (GPU/NPU) is certainly important, but without a flat, wide track (high-performance network) and a pit crew capable of sub-second tire changes and refueling (high-performance storage), even the best car cannot win. In this chapter, we will jointly build this "information superhighway" for large models and set up the "super logistics base" that enables "resuming a run after a break." We will delve into the core technology selection of RDMA networks, decode the "communication language" of distributed training -- collective communications -- and confront the biggest storage nightmare of the large model era: the optimization challenge of Checkpoints (resuming training after interruption).

3.1 RDMA Networking in Practice: InfiniBand vs. RoCE v2 -- Selection and Configuration

In traditional data centers, network communication relies on the TCP/IP protocol stack. While this protocol stack is universal and reliable, it is too bloated and inefficient for AI training scenarios that demand extreme low latency and high throughput.

3.1.1 Why is TCP/IP Unsuitable for AI Training?

Let us use an analogy to understand how TCP/IP works: it is like a courier system with rigid bureaucracy.

1. CPU Deep Involvement (CPU Overhead): When GPU-A needs to send data to GPU-B, the data must first be copied from GPU memory to CPU memory. Then the CPU intervenes, executing a series of TCP/IP protocol stack operations (data fragmentation, adding headers, calculating checksums, etc.), and finally sends it through the network card. The receiving end repeats a similar process. Throughout this process, the CPU is both "packer" and "dispatcher," consuming valuable compute resources.
2. Multiple Memory Copies: Data is copied back and forth between GPU memory, CPU memory, and network card buffer. Each copy introduces latency.
3. Kernel Overhead: Most of the network protocol stack processing occurs in the operating system kernel. The switches between user-space applications (e.g., PyTorch) and kernel space introduce significant context-switching overhead.

In distributed training, where gigabytes of gradient data need to be exchanged per second, the accumulation of this latency and CPU overhead is fatal. It causes the GPU to spend most of its time "waiting" rather than "computing," leading to a sharp decline in MFU (Model FLOPs Utilization).

3.1.2 RDMA: The "Network Bypass" Built for Performance

RDMA (Remote Direct Memory Access) technology was created precisely to solve the problems described above. It allows the network card of one host to directly read or write the memory of another host, completely bypassing the CPUs and operating system kernels of both ends.

Continuing the courier analogy, RDMA is like establishing a private, point-to-point "pneumatic tube transport system" between the memories of two hosts.

• Kernel Bypass: The control and execution of data transfer happen entirely in user space, without entering the kernel, eliminating context-switch overhead.
• CPU Offload: The entire data transfer process is handled by an RDMA-capable intelligent network card (RNIC). The CPU is completely freed to focus on other compute tasks.
• Zero-Copy: Data can be transferred directly from the sender's GPU memory (via GPUDirect RDMA technology) to the receiver's GPU memory, avoiding useless copies between CPU memories.

The effect of RDMA is significant: network latency drops from milliseconds (ms) with TCP/IP to microseconds (us), bandwidth is fully utilized, and CPU usage is greatly reduced. For large-scale AI clusters, adopting an RDMA network is usually a "necessity."

Currently, there are two main technical approaches to implementing RDMA: InfiniBand and RoCE v2.

3.1.3 InfiniBand (IB): The "Private High-Speed Rail" Built for High Performance

InfiniBand is an independent network protocol designed from the ground up for High Performance Computing (HPC) and AI.

Architecture Characteristics

• Independent Technical System: IB has its own complete technology stack, including dedicated intelligent network cards (called HCAs, Host Channel Adapters), dedicated IB switches, dedicated cables, and connectors. It is a completely independent network, isolated from Ethernet.
• Lossless by Design: IB networks employ a credit-based flow control mechanism at the link layer. Before sending data, the sender must first obtain "credit" from the receiver, confirming that the receiver has sufficient buffer space. This fundamentally prevents packet loss due to congestion, making IB networks inherently lossless.
• Centralized Management: A key component in an IB network is the Subnet Manager (SM), typically running on a switch or a dedicated server. It is responsible for discovering the network topology, assigning addresses (LIDs), computing routing tables, and distributing them to all switches, achieving centralized management and global optimization of the network.

Selection Considerations

• Advantages:
  • Extreme Performance: Provides the lowest latency (end-to-end can be below 1 us) and highest effective bandwidth in the industry. Due to its lossless nature and simple protocol stack, performance is very stable and predictable.
  • Mature and Stable: As the traditional king of the HPC field, IB technology is very mature, with well-established drivers and management tools. Deployment and operations are relatively straightforward, offering a good "out-of-the-box" experience.
• Disadvantages:
  • High Cost: IB network cards, switches, and cables are far more expensive than Ethernet equipment of the same speed.
  • Closed Ecosystem: Primarily dominated by NVIDIA (after acquiring Mellanox), with few supplier choices.
  • Independent Operations: Requires maintaining a separate IB network, demanding new skills from network engineers.

Practical Configuration and Troubleshooting

Key Components: Ensure each compute node has the Mellanox OFED driver installed. Ensure the Subnet Manager (e.g., OpenSM) is running properly.

Common Commands:

• ibstat: Check the status and port information of HCA cards.
• ibstatus: View the topology and status of the entire IB subnet.
• ibping: Test connectivity and latency between two hosts on the IB network.
• ib_write_bw / ib_write_lat: Precisely test point-to-point bandwidth and latency.

Common Issues: Port status is not Active, ibping fails (check SM and physical connections), performance is below expectations (check firmware version, PCIe speed).

3.1.4 RoCE v2: RDMA "Grafted" onto Ethernet

RoCE v2 (RDMA over Converged Ethernet v2) is a solution that carries RDMA technology over traditional Ethernet. The v2 version is based on UDP/IP, allowing it to be routed across Layer 3 networks.

Architecture Characteristics

• Based on Ethernet: RoCE uses standard Ethernet network cards (with RoCE capability) and Ethernet switches, allowing it to integrate with existing data center Ethernet networks.
• Lossless by Configuration: This is RoCE v2's most critical and complex characteristic. Ethernet is inherently "lossy" -- it drops packets when congested. To make RDMA work over it, Ethernet must be transformed into a "lossless network." This typically requires switches that support and are correctly configured with two key technologies:
  • PFC (Priority-based Flow Control, IEEE 802.1Qbb): Provides priority-based flow control. RoCE traffic can be assigned a high priority. When a switch detects that a queue for this priority is about to become congested, it sends a PAUSE frame upstream, pausing the transmission of that priority traffic, thus avoiding packet loss.
  • ECN (Explicit Congestion Notification, IETF RFC 3168): Provides explicit congestion notification. When a switch's queue buffer exceeds a certain threshold, it marks the header of forwarding packets with a "congestion experienced" flag. Upon receiving such packets, the terminal network card proactively reduces its sending rate, thereby alleviating network congestion. PFC and ECN are typically used together.

Selection Considerations

• Advantages:
  • Cost-Effective: Can leverage the mature, open, and highly competitive Ethernet ecosystem; equipment costs are relatively lower than IB.
  • Network Convergence: The AI compute network, storage network, and management network can be unified on a single Ethernet fabric, simplifying network architecture and management.
• Disadvantages:
  • Complex Configuration: Correctly configuring end-to-end lossless Ethernet is a significant challenge. All switches must support and correctly configure PFC/ECN. A single misconfiguration can lead to network packet loss, causing RDMA performance to plummet or even break.
  • Performance Sensitivity: RoCE v2 performance is more sensitive to network conditions (e.g., congestion, jitter) than IB, making troubleshooting more complex.
  • "Pseudo-Lossless" Risk: If PFC is misconfigured, it can lead to more severe problems like "deadlock."

Practical Configuration and Troubleshooting

Core: Switch-side configuration is paramount. RoCE traffic needs dedicated priority-group configuration, PFC and ECN must be enabled, and queue buffer thresholds must be finely tuned.

Server Side: Correct network card drivers must be installed, and RoCE traffic must be tagged with the correct DSCP/PCP priority values to match the switch's PFC policy.

Troubleshooting Tools: In addition to RDMA tools like rping, you must heavily rely on the switch's command-line interface to inspect PFC PAUSE frame statistics, ECN marking statistics, queue drop counts, etc., to determine if the lossless network is functioning correctly.

3.1.5 Selection Conclusion: InfiniBand vs. RoCE v2

Advice for AI Infra Engineers: If you are building a dedicated AIDC of several hundred to several thousand cards, focused primarily on training tasks, and budget allows, InfiniBand is the safer, more performant choice. If your scenario is hyperscale (tens of thousands of cards) or requires deep integration with a massive existing Ethernet infrastructure, RoCE v2 is the more scalable and cost-effective solution. However, you must invest sufficient technical resources to master the complex configuration of lossless Ethernet.

3.2 Collective Communications Basics: Analyzing Communication Primitives in NCCL (NVIDIA) and HCCL (Huawei)

After building the RDMA superhighway, we need to establish efficient "traffic rules" for the "convoy" (GPU/NPU) traveling on it. This is the role of Collective Communications. In distributed training, individual GPUs performing independent calculations is meaningless. They must work as a whole, frequently exchanging and synchronizing data (primarily gradients).

Collective communication libraries, such as NVIDIA's NCCL and Huawei's HCCL, are the implementers of these "traffic rules." They provide highly optimized communication operation functions -- called communication primitives -- tailored for specific hardware and network topologies.

3.2.1 Why is Collective Communication Needed?

Take the most common Data Parallelism training as an example:

1. Distribution: At the start of training, model parameters need to be distributed from the main node (Rank 0) to all participating GPUs.
2. Computation: Each GPU receives a mini-batch of data and independently computes gradients.
3. Aggregation: This is the most critical step. The gradients from all GPUs must be aggregated (typically averaged) to update the global model.
4. Update: All GPUs use the aggregated gradients to synchronously update their own model parameters.

If a naive "parameter server" model is used for gradient aggregation (i.e., all GPUs send gradients to a central node, which computes the average and sends it back), this central node quickly becomes a bottleneck. The efficiency of training would sharply decrease as the number of GPUs increases.

Collective communications, through clever algorithms (e.g., Ring, Tree), distribute the communication load evenly across all nodes, avoiding a central bottleneck and enabling efficient global data exchange.

3.2.2 Core Communication Primitives Explained

Here are the core communication primitives that every AI Infra engineer must understand:

Broadcast

• Function: One-to-many. Copies data from one node (typically the root node) and distributes it to all other nodes in the group.
• AI Scenario: At the start of training, broadcast the initialized model weights from Rank 0 to all GPUs, ensuring everyone starts from the same point.
• NCCL/HCCL Implementation: Typically uses a tree algorithm. The root node sends to two child nodes, each child node sends to its two children, and so on, achieving logarithmic time complexity distribution.

Reduce

• Function: Many-to-one. Aggregates data from all nodes in the group using a specified operation (e.g., SUM, AVG, MAX) and stores the result on the root node.
• AI Scenario: Collect loss values from all GPUs, compute the average loss on the main node, and print it.
• NCCL/HCCL Implementation: Also uses a tree algorithm. Leaf nodes send data and compute upward to their parent nodes, level by level, until reaching the root node.

All-Reduce

• Function: This is the most core, most frequent, and most expensive primitive in distributed training. It performs a reduce operation (e.g., SUM) on data from all nodes in the group, then broadcasts the final result back to all nodes. It is equivalent to a Reduce operation plus a Broadcast operation.
• AI Scenario: In data-parallel training, each GPU computes its own gradients. All gradients must be summed (SUM) and averaged, with the final averaged gradient distributed to every GPU for updating its model.
• NCCL/HCCL Implementation: This is the primary focus of optimization. The most classic algorithm is Ring-AllReduce.
  • Reduce-Scatter Phase: Imagine all GPUs arranged in a ring. The data is divided into N chunks (N = number of GPUs). In the first step, GPU i sends its i-th chunk to the next GPU i+1, while receiving the i-1-th chunk from the previous GPU i-1, and adds the received data to its local corresponding chunk. This process repeats N-1 times, with different data chunks each time. After N-1 steps, each GPU i holds the i-th chunk of the final result (i.e., the sum of the i-th chunks from all GPUs).
  • All-Gather Phase: N-1 ring passes are performed again. This time, each GPU passes the final result chunk it has already computed to all other GPUs in the ring. After N-1 steps, every GPU holds the final result for all chunks -- the complete global gradient sum. The elegance of Ring-AllReduce is that at any given moment, all GPUs and their network links are kept busy, maximizing bandwidth utilization. Besides Ring, NCCL/HCCL intelligently selects more complex algorithms (like Double Binary Tree) based on network topology and data size.

All-Gather

• Function: Many-to-many. Collects data from each node in the group, concatenates it into a large tensor, and distributes it to all nodes.
• AI Scenario: In Tensor Parallelism or Pipeline Parallelism, model weights or activations are sharded across different GPUs. At some point during computation, a specific GPU may need the complete weights or activations, requiring an All-Gather operation.

3.2.3 NCCL and HCCL: The Ecosystem Implementers

NCCL (NVIDIA Collective Communications Library)

NVIDIA's deeply optimized collective communication library for its own GPUs and networks (NVLink, InfiniBand).

Topology Awareness: NCCL can automatically detect hardware topology. For example, inside an 8-GPU HGX node, it will prioritize using the fastest NVLink for ring communication. When cross-node communication is needed, it seamlessly switches to the RDMA network (IB or RoCE).

Debugging and Optimization: AI Infra engineers must learn to use NCCL's debugging environment variables:

• NCCL_DEBUG=INFO: Prints NCCL initialization information, selected communication algorithm (Ring/Tree), detected network topology, etc. This is the starting point for troubleshooting.
• NCCL_DEBUG=WARN: Prints only warnings and errors, used for production monitoring.
• NCCL_ALGO=Ring / Tree: Forces NCCL to use a specific algorithm for performance comparison and debugging.
• NCCL_PROTO=Simple / LL128: Adjusts the communication protocol, impacting performance.
• NCCL_P2P_LEVEL: Controls the scope of P2P communication (e.g., using NVLink only within a node).

Common Issues: NCCL Timeout is the most common error, typically indicating network congestion, packet loss, or a GPU hang breaking the communication ring.

HCCL (Huawei Collective Communication Library)

Huawei's Ascend ecosystem counterpart to NCCL, deeply optimized for the DaVinci architecture and Huawei's self-developed HCCS on-chip interconnect and RoCE network.

Shared Principles: HCCL also implements core primitives like All-Reduce, and the underlying algorithmic concepts (Ring/Tree) are the same as NCCL's.

Ecosystem Adaptation: PyTorch for Ascend and MindSpore frameworks call HCCL to handle communication for distributed training.

Debugging: HCCL also provides similar environment variables (e.g., ASCEND_GLOBAL_LOG_LEVEL) to control log output and help locate communication issues.

Advice for AI Infra Engineers: Collective communication is the soul of distributed training. When you encounter "adding machines actually decreases performance" or "training logs get stuck somewhere," suspect a collective communication problem first. Learning to read NCCL/HCCL INFO logs -- understanding what algorithm was chosen, what topology was detected -- is an essential skill for diagnosing these kinds of "mysteries."

3.3 Storage Challenge: High-Concurrency Read/Write Optimization for Checkpoints

In large model training that can take weeks or even months, an unexpected interruption (hardware failure, software bug, power outage) can render the computation since the last checkpoint useless. The Checkpoint mechanism is therefore the most important safeguard. It periodically saves the complete state of the model (all weights, optimizer state, learning rate, etc.) to persistent storage.

However, as model parameters explode (from tens of billions to trillions), the size of checkpoint files has also grown from tens of GB to several TB. This presents unprecedented challenges to the storage system.

3.3.1 The "Double Nightmare" of Checkpoints

Write Nightmare: Training Paused, Wasting Time

Traditional checkpoint operations are synchronous: training must completely pause and wait for all data to be written to persistent storage before it can continue.

A 70B (70 billion parameter) FP16 model has a checkpoint size of approximately 70B * 2 (weights) + 70B * 2 * 2 (Adam optimizer states) = 420 GB. With optimization strategies like ZeRO, this state may also be sharded across all GPUs.

When hundreds of GPUs simultaneously initiate write requests for this 420 GB file to the storage system, if the storage system is not up to the task, this "pause" can last several minutes or even tens of minutes.

Cost Analysis (computational example): Assume a thousand-card cluster checkpoints once per hour, with each write taking 10 minutes. This means the cluster spends 1/6 of its time (approximately 16.7%) waiting for storage -- a substantial compute investment wasted. This directly lowers MFU and is a major killer of AIDC operational efficiency. Actual figures depend on storage bandwidth, checkpoint size, and frequency, and should be measured in your own environment.

Read Nightmare: Resume Training, Long Wait

When training needs to resume from a checkpoint after an interruption, the same large-scale concurrent read problem arises. Hundreds of GPUs need to read TB-level model state files from the storage system simultaneously and load them into their respective GPU memories. If this process is slow, it also lengthens Mean Time To Recovery (MTTR), reducing the cluster's effective usage.

3.3.2 Why Do Traditional NAS (e.g., NFS) Systems Crash?

Many teams initially use general-purpose NAS (Network Attached Storage, like NFS) for convenience to store checkpoints. But this quickly runs into bottlenecks:

• Metadata Bottleneck: When hundreds of clients are creating/writing files simultaneously, all metadata operations (like file name, permissions, size updates) are directed at a single NFS server. Contention for metadata locks becomes extremely fierce, causing slow system response.
• Single Point Bandwidth Bottleneck: No matter how high the NFS server is configured, it remains a single point. Its network bandwidth and disk I/O capacity are limited and cannot handle the concurrent write flood from hundreds of high-performance nodes.

3.3.3 The Solution: Parallel File Systems and Optimization Strategies

To solve the checkpoint problem, a parallel file system designed for large-scale concurrent reads and writes is essential.

Core Idea: Divide and Conquer

Parallel file systems separate metadata management from data storage.

• Metadata Server (MDS/MGS): Dedicated to handling metadata requests, such as file names, directory structures, and file permissions.
• Data Servers (OSS/OSD): Numerous data servers are responsible for storing the actual file content. A large file is "striped" (sliced) and stored in parallel across multiple data servers, similar to RAID 0.

When a client wants to write a file, it first queries the MDS ("Which OSSes should I write the data to?"), and then can establish connections directly and in parallel with multiple OSSes, writing different slices of the data simultaneously. This fundamentally breaks the single-point bottleneck.

Mainstream Parallel File Systems:

• Lustre: The king of the open-source community, widely used in the world's TOP500 supercomputing centers. Powerful, excellent performance, but configuration and tuning have a certain learning curve.
• GPFS (IBM Spectrum Scale): A mature commercial solution with comprehensive features and rich policy support, offering commercial support, but with higher cost.
• BeeGFS, OrangeFS: Other excellent open-source parallel file systems.

3.3.4 Practical Checkpoint Optimization Strategies

Even when using a parallel file system, fine-grained optimization needs to be combined with the upper-layer application to minimize the impact of checkpoints.

Hardware Level

• NVMe for Metadata: MDS is key to performance and must use ultra-low-latency NVMe SSDs.
• Separate Data and Metadata Networks: Ensure low latency for metadata requests.
• Hybrid Storage for OSS: Depending on cost and performance requirements, use NVMe or SATA SSDs as the hot tier for OSS, and HDDs as the cold tier.

File System Tuning

• Stripe Tuning: This is the most important tuning parameter for parallel file systems.
  • stripe_count: How many slices a file is cut into, distributed across how many OSSes.
  • stripe_size: The size of each slice.
  • Tuning Principle: For sequential writes of large files like checkpoints, set a larger stripe_size (e.g., 4 MB or larger) and a stripe_count equal to the number of OSSes, to maximize the aggregated bandwidth of all data servers.

Application and Framework Level Optimization

• Asynchronous/Non-blocking Checkpoint: This is the "nuclear weapon" for reducing training pause time.
  • User-Space Implementation: In the training framework, when a checkpoint is needed, the main training process hands over the pointer to the model state to a dedicated "I/O subprocess" or "I/O thread pool," and then immediately returns to continue training. The background I/O process is responsible for slowly writing the data to persistent storage. This requires careful handling of memory management and data consistency.
  • Framework Support: More and more training frameworks (e.g., DeepSpeed, Megatron-LM) are starting to natively support or experiment with non-blocking checkpoint functionality.
• Two-Phase Checkpoint:
  • Phase 1 (Fast Dump): Training pauses. The model state from all GPU memories is written in parallel to the local high-speed NVMe SSD of each node as fast as possible. This process is very fast because it involves local writes, no network. Training can resume quickly.
  • Phase 2 (Background Archival): An independent background script or service is responsible for collecting the checkpoint shards from the local NVMe SSDs of all nodes, aggregating them, and uploading them to the final, cheaper persistent storage (e.g., parallel file system or object storage).
• Incremental and Differential Checkpoints: For scenarios like SFT (Supervised Fine-Tuning), model weight changes might be small. You can save only the parts that changed compared to the last checkpoint, but this is more complex to implement.
• Sharded Optimizer State Saving: Optimizer states like Adam occupy the majority of checkpoint space. Consider reducing their saving frequency, or using a memory-efficient optimizer (e.g., Adafactor).

Advice for AI Infra Engineers: Checkpoint optimization is a system engineering task. You need to collaborate with algorithm engineers, framework developers, and storage experts. Your job is not just deploying a Lustre system; it is also delving into the training code to promote the implementation of advanced strategies like two-phase checkpoint or asynchronous checkpoint. In your Grafana dashboard, there should be a dedicated panel monitoring the duration of each checkpoint, making this metric one of the AI Infra team's core KPIs.