祝融说。

In the first three chapters, we have built a complete top-down understanding: we comprehended the extreme thirst for computing power in large AI models, became familiar with the software stack for harnessing that power, and deeply analyzed the micro-architecture of the GPU chip -- the source of all computing power. Now we possess the most powerful engine (the GPU) and know how to drive it (the software stack). The task that remains is to design and build a supercar around this engine -- a GPU server that is balanced, stable, reliable, and capable of unleashing its ultimate potential.

A GPU server is far more than simply plugging a few GPU cards into an ordinary server motherboard. Especially in the era of large models, a top-tier GPU server is itself a precision instrument that condenses the pinnacle of insight from systems engineering, high-speed signaling, thermal design, and power management. Its design has only one goal: to eliminate every possible bottleneck and ensure that 8 or even 16 expensive GPUs can work together at maximum efficiency, around the clock, without interruption.

In this chapter, we will take the industry-recognized "gold standard" and reference design -- the NVIDIA DGX series -- as our core object of study. Through a meticulous analysis of the DGX server, we will learn the design essence of a top-tier GPU server. Although the chapter outline uses the DGX A100 as its example, its design philosophy and core architecture are equally essential for understanding the latest DGX H100. We will explain both side by side, allowing readers to perceive the thread of their evolution.

We will first get to know the NVIDIA DGX, understanding why it is not merely a server but an "AI supercomputer in a box" that integrates hardware, software, and support services. Next, we will dive into the overall design of the DGX A100 and glimpse its internal workings. Then, like dissecting precision machinery, we will analyze its four key subsystems one by one: the CPU and memory subsystem, the PCIe subsystem, the NVLink subsystem, and the other auxiliary subsystems (such as storage, network, power, and cooling).

By the end of this chapter, you will be able to answer the following questions: Why do GPUs need such powerful CPUs and such massive memory? What key role does the PCIe switch play? How do NVLink and NVSwitch build the high-speed highway between GPUs? How do we supply stable, redundant power to these "power hogs" and carry away the enormous heat they generate? Ultimately, you will possess the comprehensive, system-level vision needed to evaluate, design, and understand a high-performance GPU server.

4.1 First Encounters with NVIDIA DGX

Before delving into technical details, it is necessary to understand the positioning and value of the NVIDIA DGX. The DGX is not a mere hardware brand; it is a full-stack AI infrastructure solution that NVIDIA provides to enterprises and research institutions, designed to work "out of the box."

4.1.1 The Genesis of DGX: Eliminating the Complexity of AI Infrastructure

In the early days of the deep learning boom, enterprises and research institutions faced enormous challenges when building their own GPU computing platforms:

• Confusion in Hardware Selection: Which GPU should be chosen? Which CPU should be paired with it? How much memory is needed? Are the motherboard and chassis compatible? How should networking and storage be configured? These questions were riddled with "pitfalls."
• The Nightmare of System Integration: Assembling the hardware was only the first step. Next came installing the operating system, GPU drivers, CUDA toolkit, cuDNN, and a host of other low-level software. Version compatibility issues, driver conflicts, and performance tuning could consume weeks or even months.
• A Missing Software Stack: Even if the hardware and drivers were running, configuring the container environment (such as Docker), the resource scheduler (such as Slurm), the mainstream AI frameworks (TensorFlow, PyTorch), and all their dependencies and optimizations was yet another enormous undertaking.
• No Unified Support: When problems arose, responsibility was hard to pin down. Was it a hardware fault (the GPU? the CPU? the motherboard?), a driver problem, or a framework bug? Users were forced to be shuttled back and forth between multiple vendors.

NVIDIA keenly perceived these pain points and realized that to drive the adoption of AI, it needed to provide a "turnkey" solution that would free data scientists and researchers from the drudgery of infrastructure setup and allow them to focus on algorithms and models themselves. Thus the DGX was born.

4.1.2 The DGX Value Proposition: "An AI Supercomputer in a Box"

The value of the DGX can be summarized as follows:

1. An Optimally Tuned Hardware Platform: The hardware design of the DGX server represents NVIDIA's best practices, at a given point in time, for system-level integration of its latest GPUs. From motherboard topology and signal integrity to power distribution and cooling airflow, every detail has been meticulously engineered and rigorously validated, with the aim of maximizing the overall performance and stability of a multi-GPU system. It is a "pre-tuned," balanced system, not a simple accumulation of hardware.
2. A Fully Integrated Software Environment: When you buy a DGX, what you receive is more than just hardware. Each DGX ships with DGX OS preinstalled -- an optimized, Ubuntu-based operating system containing all the required drivers and low-level software. More importantly, NVIDIA provides access to the NGC (NVIDIA GPU Cloud) software repository. NGC contains a vast collection of Docker containers that are pre-built, tested, and optimized, covering all the mainstream AI frameworks, HPC applications, data science tools, and more. With a single docker pull command, users obtain a ready-to-use development environment with optimal performance.
3. Enterprise-Grade Support and Service: NVIDIA provides end-to-end enterprise support to DGX customers. Whether the issue is hardware, software, or performance, users only need to contact NVIDIA's expert team to receive a one-stop solution. This "single point of responsibility" service model gives enterprises confidence in deploying AI applications at scale in production.
4. A Scalable Reference Architecture: The DGX is not merely a single server. NVIDIA also provides the DGX POD and DGX SuperPOD reference architectures, built on DGX and high-performance networks (such as InfiniBand). These architectures offer proven design blueprints for constructing AI supercomputers ranging from dozens to tens of thousands of cards, covering every aspect including compute, networking, storage, and software management.

4.1.3 The Evolution of the DGX Family

Since the first-generation DGX-1 was released in 2016, the DGX series has evolved in step with the generational turnover of NVIDIA's GPU architectures:

DGX-1: Based on the Pascal P100 GPU, it first introduced an 8-GPU hybrid cube-mesh topology interconnected via NVLink.

DGX-2: Based on the Volta V100 GPU, it was a behemoth housing 16 V100 GPUs. It first introduced the NVSwitch chip, achieving a fully connected NVLink Fabric among the 16 GPUs with total bandwidth of 2.4 TB/s, making it the most powerful single-unit AI server of its time.

DGX A100: Based on the Ampere A100 GPU, it returned to a more compact 8-GPU design. It adopted third-generation NVLink and second-generation NVSwitch, with total bidirectional bandwidth between the 8 GPUs reaching 4.8 TB/s. This chapter will primarily analyze its design.

DGX H100: Based on the Hopper H100 GPU, it is the current flagship. It likewise uses an 8-GPU design but has been upgraded to fourth-generation NVLink and third-generation NVSwitch, further raising the total bidirectional bandwidth between the 8 GPUs to 7.2 TB/s. At the same time, it shares a design lineage with the Grace Hopper superchip, showing the future trend toward heterogeneous convergence.

From this preliminary acquaintance with the DGX, we understand that it is not only a hardware product but also a design philosophy and an ecosystem. Next, we will go inside this "AI box" to see what truly lies within.

4.2 The Overall Design of the NVIDIA DGX A100

The DGX A100 server is a standard 6U rack-mounted server whose internal structure is compact and precision-engineered; all components are laid out in a modular fashion to facilitate maintenance and upgrades. Lifting its top cover reveals a clean layout designed to maximize both performance and cooling efficiency.

4.2.1 Physical Layout and Core Modules

The interior of the DGX A100 can be divided into several key functional areas:

1. GPU Baseboard: This is the core and soul of the entire server. A large, multi-layer PCB spans the middle of the chassis, carrying 8 A100 SXM4 modules and 6 NVSwitch chips. These 8 GPUs and 6 NVSwitches together constitute a fully connected NVLink Fabric.
2. CPU Modules and Memory: At the front of the chassis (near the cold aisle) sit the CPU compute modules. The DGX A100 is equipped with 2 AMD EPYC 7002-series CPUs (such as the 7742, with 64 cores). DIMM memory slots are packed around each CPU, supporting up to 2 TB of DDR4 memory in total.
3. Networking and PCIe Expansion Area: At the rear of the chassis (near the hot aisle) is the I/O expansion area. It contains:
   1. 8 Mellanox ConnectX-6 200 Gb/s InfiniBand/Ethernet NICs, dedicated to high-speed distributed training communication between GPUs (GPUDirect RDMA).
   2. 1 dual-port Mellanox ConnectX-6 200 Gb/s NIC for the storage network.
   3. 1 management network port and multiple USB ports.
   4. 2 NVMe U.2 SSDs, used as high-speed OS drives and cache.
4. Power and Cooling Systems:

Power: Beneath the rear of the chassis are multiple (typically 6) hot-swappable redundant power supply units (PSUs), providing stable, reliable power to this beast, which can draw up to 6.5 kW.

Cooling: The entire chassis interior is designed as a strict front-to-back airflow path. The CPU and memory area at the front is cooled by one bank of fans, while the GPU and NVSwitch region in the middle -- the largest "heat source" -- is cooled by another, more powerful bank of fans. The GPU modules themselves are also covered by large passive heatsinks.

4.2.2 Logical Architecture: A Data-Centric Design

Logically, the DGX A100's architecture exemplifies the "fat GPU node" design philosophy. Its core topology can be summarized as follows:

• The GPU Cluster at the Center: The 8 A100 GPUs form a tightly coupled compute core through the NVSwitch. The bandwidth for communication among them (4.8 TB/s) far exceeds their bandwidth for communicating with the rest of the system.
• Dual CPUs as "Waiters": The 2 powerful AMD EPYC CPUs handle all the "odd jobs" -- running the operating system, data preprocessing, task scheduling, network I/O processing, and so on. Each CPU connects to 4 GPUs and 4 high-speed NICs through the PCIe bus, forming a relatively independent NUMA domain.
• PCIe Switches as "Interchanges": To connect the CPU to the numerous GPUs, NICs, and other PCIe devices, the DGX A100 uses multiple PCIe Switch chips internally. These switches play a key role in expanding the number of PCIe lanes and optimizing data paths.
• Separated Network Planes: The DGX A100 clearly distinguishes networks serving different purposes:
  • Compute Fabric: 8 dedicated 200G NICs, used primarily for gradient synchronization (All-Reduce) during multi-node distributed training.
  • Storage Fabric: 1 dedicated 200G NIC, used to connect to external high-performance parallel file systems or object storage.
  • In-Band/Out-of-Band Management Network: used for server management and monitoring.

This clear, GPU-centric design ensures that data flows through the system in the most efficient manner possible. Whether between GPUs, between GPUs and CPUs, or between GPUs and the external network/storage, dedicated high-bandwidth channels exist, avoiding resource contention and bottlenecks.

4.3 Designing the CPU and Memory Subsystem of the NVIDIA DGX A100

In a GPU server, although the CPU is not the primary source of compute power, it serves as the system's "brain" and "chief housekeeper," and its performance and configuration directly determine the height of the whole system's short board. A weak CPU subsystem will leave the GPU in a state of prolonged "hunger," idly waiting, and cause a tremendous waste of computing power.

4.3.1 Why Do We Need a Powerful CPU?

In the AI training workflow, the CPU mainly shoulders the following key tasks:

Data Loading and Preprocessing

The first step in AI training is reading massive amounts of raw data (such as images and text) from the storage system. This process is itself I/O-intensive.

Raw data often cannot be fed into the GPU directly. It must undergo a series of preprocessing operations, such as image decoding, cropping, resizing, rotation, and normalization. These operations are typically carried out on the CPU by multiple parallel "data loader" worker processes.

If the CPU's processing speed cannot keep pace with the speed at which the GPU consumes data, the GPU will sit idle, and training efficiency will suffer greatly. This is the so-called "Input Pipeline Bottleneck." A CPU with more cores and a higher clock speed can support more parallel data loader processes, and thus deliver higher data throughput.

Operating System and Task Scheduling

The CPU is responsible for running the Linux operating system, the DGX OS software stack, the resource scheduler (such as Slurm), and the user's Python main program.

Network I/O Processing

In distributed training, the CPU must handle the network protocol stack, package and unpack data, and transfer data between the NIC and memory over the PCIe bus. Although technologies such as GPUDirect RDMA can bypass the CPU, CPU involvement remains unavoidable in many scenarios.

Executing Some AI Operators

Although the vast majority of computation runs on the GPU, there are always some operators that are unsuited to the GPU, or for which the framework has not yet implemented a GPU version, and these execute on the CPU.

4.3.2 The DGX A100's CPU Selection: AMD EPYC

The DGX A100 chooses to be equipped with 2 AMD EPYC 7742 CPUs. This choice is based on a number of considerations:

An Extremely High Core Count: Each EPYC 7742 has 64 physical cores, and two of them form a powerful computing platform with 128 cores/256 threads. With so many cores, large-scale parallel data preprocessing tasks can be handled with ease.

Abundant PCIe 4.0 Lanes: This is the most critical factor. Each AMD EPYC 7002-series CPU natively supports 128 PCIe 4.0 lanes, and two CPUs provide 256 lanes. This massive amount of I/O bandwidth is the fundamental guarantee for connecting 8 GPUs, 8 high-speed NICs, and numerous other devices. By contrast, the Intel Xeon CPUs of the same era were at a disadvantage in the number of PCIe lanes they could offer.

High Memory Bandwidth and Capacity: Each EPYC CPU supports 8 memory channels, and the entire system supports 16 channels of DDR4-3200 memory with a maximum capacity of 2 TB. This provides ample room for the temporary staging and preprocessing of large volumes of data.

4.3.3 The NUMA Architecture and CPU-GPU Affinity

In a dual-CPU system such as the DGX A100, the memory access architecture is NUMA (Non-Uniform Memory Access). This means that each CPU accesses memory directly attached to itself ("local memory") at the highest speed, whereas accessing memory attached to the other CPU ("remote memory") requires crossing the interconnect bus between the CPUs (such as AMD's Infinity Fabric), incurring higher latency.

This NUMA architecture is of critical importance to GPU server performance and demands strict affinity binding:

• Topology Design: The DGX A100's design is balanced. CPU 0 connects via the PCIe bus to GPUs 0-3 and 4 NICs; CPU 1 connects to GPUs 4-7 and the other 4 NICs.
• Software Binding: At the software level, it must be ensured that:
  • Processes handling tasks related to GPUs 0-3 (such as data loaders) are pinned to the physical cores of CPU 0.
  • The memory allocated by these processes is preferentially obtained from CPU 0's local memory.
  • When GPUs 0-3 need to send data over the network, the 4 NICs connected to CPU 0 are used preferentially.
• Purpose: Through this strict affinity binding, the vast majority of data transfers can be kept within NUMA nodes, avoiding costly cross-NUMA-node access and thereby maximizing I/O performance. DGX OS and NVIDIA's software stack have already handled these complex binding tasks for the user.

4.3.4 System Memory: Is Bigger Always Better?

The DGX A100 supports up to 2 TB of system memory. Such a large amount of memory serves mainly to:

• Cache Datasets: For datasets that can be loaded entirely into memory, reading them into system memory in a single pass can dramatically accelerate subsequent training, avoiding the need to read from slow external storage at every epoch.
• Buffer Data Preprocessing: Data loaders require large amounts of memory as buffers to hold batches of preprocessed data, ready for the GPU to consume at any moment.
• Support Emerging Technologies (such as ZeRO-Offload): Microsoft DeepSpeed's ZeRO-Offload technology can "offload" model parameters or optimizer states that do not fit in GPU memory down to the CPU's system memory. This requires a vast amount of CPU memory to serve as an "extension of the VRAM."

In short, although the CPU and memory subsystem is not the protagonist, its powerful multi-core performance, abundant PCIe lanes, and enormous memory capacity form the solid foundation that supports the efficient operation of a GPU cluster.

4.4 Designing the PCIe Subsystem of the NVIDIA DGX A100

PCI Express (PCIe) is the standard bus in modern servers that connects the CPU to various high-speed peripherals (such as GPUs, NICs, and NVMe SSDs). In a GPU server, the quality of the PCIe subsystem design directly determines whether data can flow smoothly among the CPU, GPU, network, and storage.

4.4.1 PCIe 4.0: Doubled Bandwidth

The DGX A100 is one of the first servers to fully embrace the PCIe 4.0 standard. Compared with PCIe 3.0, PCIe 4.0 doubles the bandwidth per lane, from about 1 GB/s to about 2 GB/s.

A typical PCIe x16 slot can reach bidirectional bandwidth of 64 GB/s under PCIe 4.0, compared with only 32 GB/s under PCIe 3.0.

This doubling of bandwidth is of critical importance to GPUs and high-speed NICs, which must exchange large quantities of data with the CPU, effectively alleviating the communication bottleneck between the CPU and the GPU.

4.4.2 The Problem of "Too Many Demands, Too Few Lanes"

Although the DGX A100 is equipped with a powerful CPU offering 256 PCIe 4.0 lanes, the number of devices to connect is simply enormous:

• 8 A100 GPUs, each requiring an x16 connection. 8 * 16 = 128 lanes.
• 8 ConnectX-6 compute NICs, each also requiring an x16 connection. 8 * 16 = 128 lanes.
• 1 ConnectX-6 storage NIC, requiring an x16 connection. 1 * 16 = 16 lanes.
• 2 NVMe SSDs, each requiring an x4 connection. 2 * 4 = 8 lanes.

There are also the southbridge chip, the management controller, and other devices.

The total demand far exceeds the 256 lanes the CPU can supply. How is this problem solved? The answer is to introduce PCIe switches.

4.4.3 The Key Role of the PCIe Switch

A PCIe switch, much like a network switch, has an upstream port that connects to the CPU (or to another switch) and multiple downstream ports that connect to endpoint devices. Its functions are:

1. Fan-out: A single switch can take one x16 upstream link and fan it out into multiple x16, x8, or x4 downstream links.
2. Peer-to-Peer (P2P) Communication: This is the most important function of a PCIe switch in a GPU server. Two devices attached to the same PCIe switch (for example, two GPUs, or one GPU and one NIC) can perform direct peer-to-peer data transfers without passing through the CPU or system memory. This dramatically reduces communication latency and frees up the CPU and memory bandwidth.

4.4.4 The PCIe Topology of the DGX A100

The PCIe topology of the DGX A100 is an ingenious masterpiece, designed to maximize the efficiency of P2P communication. Its general structure is as follows:

A Dual-NUMA-Node Design: The entire system is divided symmetrically into two "domains" composed of PCIe switches, connected respectively to CPU 0 and CPU 1.

A Three-Layer Switching Structure:

1. Layer 1 (Direct CPU Connection): Each CPU uses its 128 PCIe lanes, splitting them into multiple x16 links that connect to multiple Layer 2 PCIe switches.
2. Layer 2 (Main Switches): The DGX A100 motherboard integrates several large PCIe 4.0 switch chips (such as the Broadcom PEX series). The 4 GPUs and 4 compute NICs attached to each CPU are cleverly grouped and connected to these main switches.
3. Layer 3 (Device Connection): The main switches then connect downward to the individual PCIe devices.

Carefully Designed P2P Paths:

• GPU-to-GPU P2P: The PCIe P2P communication path between any two GPUs is optimized.
• GPUDirect RDMA: This is the most critical P2P application. The DGX A100's topology ensures that each GPU falls under the "jurisdiction" of the same PCIe switch as a dedicated compute NIC. This means that when a GPU needs to send data over the network (for example, gradient synchronization in distributed training), it can use GPUDirect RDMA to let the data flow directly from the GPU's memory to the NIC and out onto the wire, with no CPU intervention and no copying through system memory. This is crucial for reducing communication latency in distributed training.
• GPUDirect Storage: Likewise, the GPU can read data directly from an NVMe SSD via PCIe P2P, bypassing the CPU to achieve higher data-loading bandwidth.

Through this complex yet efficient PCIe switching network, the DGX A100 builds a far-reaching "data interchange system" for all the high-speed devices within the server, ensuring that there is always a shortest, fastest path between any two nodes that need to communicate.

4.5 Designing the NVLink Subsystem of the NVIDIA DGX A100

If PCIe is the "national highway" that connects the GPU to the outside world, then NVLink is the dedicated, enclosed "F1 track" between GPUs, where there is no speed limit. In large-model training, both tensor parallelism and pipeline parallelism demand extremely frequent and massive exchanges of data among GPUs, and the performance of the NVLink subsystem directly determines the efficiency of model parallelism.

4.5.1 Third-Generation NVLink and the SXM4 Form Factor

Third-Generation NVLink: The A100 GPU that powers the DGX A100 supports third-generation NVLink. Each A100 GPU has 12 NVLink 3.0 links, each with bidirectional bandwidth of 50 GB/s. Thus each A100 GPU offers a total NVLink bandwidth of up to 600 GB/s.

The SXM4 Module: To accommodate such a high density of I/O interfaces, the data-center version of the A100 GPU adopts the SXM4, a dedicated mezzanine form factor, rather than the standard PCIe card form factor. The SXM4 module is mounted on the GPU baseboard through a high-density connector that integrates power delivery, management, and all the NVLink and PCIe signals.

4.5.2 NVSwitch: The Core of Building a Fully Connected Fabric

Although each GPU has 600 GB/s of bandwidth, how can 8 GPUs be connected efficiently? If one attempted pairwise direct connections, a GPU with 12 links could directly connect to only a small number of neighbors, making globally efficient communication impossible.

To solve this, NVIDIA designed the NVSwitch chip. An NVSwitch is a high-speed switch chip dedicated to routing NVLink traffic.

Second-Generation NVSwitch: The DGX A100 uses 6 second-generation NVSwitch chips. Each NVSwitch chip has multiple NVLink ports.

A Fully Connected (All-to-All) Topology: The GPU baseboard design of the DGX A100 is the crown jewel of its technology. Through these 6 NVSwitches, it cleverly achieves a fully connected topology among the 8 A100 GPUs. This means:

Between any two GPUs, there exists a direct communication path composed of 12 NVLink links, with a total bandwidth of 600 GB/s.

These 8 GPUs form a bandwidth-non-blocking NVLink Fabric, whose aggregate bidirectional bandwidth reaches 4.8 TB/s (8 * 600 GB/s / 2, divided by 2 because each connection is counted twice).

How It Works: Each A100 GPU divides its 12 NVLink links into 6 groups of 2 links, connecting each group to one of the 6 NVSwitch chips. When GPU 0 wishes to communicate with GPU 1, it sends data packets out over the links attached to one NVSwitch; the NVSwitch, acting like a network switch, forwards the packets to the links attached to GPU 1 according to the packets' destination addresses. Because every GPU is connected to all 6 NVSwitches, a path between any two GPUs is guaranteed.

4.5.3 The Value of the NVLink Subsystem

This fully connected, high-bandwidth NVLink Fabric is the fundamental reason the DGX A100 can execute large-scale model parallelism so efficiently:

• Efficient Tensor Parallelism: Tensor parallelism requires frequent All-Reduce or All-Gather operations among GPUs. The NVLink Fabric provides hardware-level ultra-low latency and ultra-high bandwidth for these collective communication operations, making tensor parallelism across as many as 8 GPUs possible.
• Smooth Pipeline Parallelism: In pipeline parallelism, the activations of each layer must be passed from one GPU to the next. NVLink likewise provides the optimal path for these point-to-point transfers.
• A Unified Programming Model: For developers, the 8 GPUs are logically just a single "giant monolithic GPU" with enormous memory and computing power. When using libraries such as NCCL for multi-GPU communication, developers need not concern themselves with the underlying physical topology, because the communication performance between any two GPUs is symmetric and consistent.

A Comparison with the DGX H100: The DGX H100 has been upgraded to fourth-generation NVLink (each H100 has 18 links, for a total bandwidth of 900 GB/s) and third-generation NVSwitch. The aggregate fully connected bandwidth of its 8 GPUs reaches 7.2 TB/s, further improving the efficiency of model parallelism.

4.6 Designing the Other Auxiliary Subsystems

The excellence of a top-tier GPU server lies in its attention to every detail. Beyond the three core subsystems of compute and communication, the auxiliary subsystems of power, cooling, storage, and management are equally vital.

4.6.1 The Storage Subsystem

OS Drives: The DGX A100 is equipped with 2 high-speed NVMe U.2 SSDs, configured as RAID 1 (mirroring), on which the DGX OS is installed and logs are stored. The NVMe protocol and its direct PCIe connection deliver extremely low latency and high throughput, ensuring smooth system startup and operation.

Internal Data Drives: The DGX A100 also offers 4 NVMe U.2 slots, which can be used to build an internal high-speed data cache pool of up to 30 TB. This is very useful for holding frequently accessed small-to-medium datasets, or for serving as a "level-1 cache" in front of external storage.

External Storage Connection: For large-model training, datasets are typically held on external parallel file systems (such as Lustre or GPFS) or object storage (such as Ceph), which consist of hundreds of hard drives. The DGX A100 connects to these external storage systems through its dedicated 200G storage NIC, gaining a continuous, high-bandwidth supply of data.

4.6.2 The Power Subsystem

High Power Consumption: A DGX A100 server running at full load can draw up to 6.5 kilowatts (kW) of peak power. This is equivalent to running three household air conditioners at the same time.

Redundant Design: To guarantee enterprise-grade reliability, the DGX A100 adopts an N+N redundant power design. It is equipped with 6 power supply modules (PSUs) of 3 kW each, divided into two groups of 3. Under normal operation, only a portion of the PSUs in each group need to supply power. Should any one PSU fail, its load is immediately taken over by the other PSUs in the same group, without bringing the server down.

High Conversion Efficiency: These PSUs are typically 80 Plus Titanium or Platinum certified for high efficiency, reducing energy loss during the conversion from alternating current to direct current and lowering the data center's PUE.

4.6.3 The Thermal Subsystem

Enormous Heat: The vast majority of the 6.5 kW of electrical power input is ultimately converted into heat. How to remove this heat effectively from the cramped 6U chassis is a huge engineering challenge.

An Air-Cooling Design: The DGX A100 adopts a sophisticated air-cooling design.

Strict Airflow Isolation: The chassis interior is divided into several independent airflow paths by baffles, ensuring that cold air flows precisely over every heat-generating component (CPU, GPU, NVSwitch, memory, PSUs) without giving rise to recirculating hot air.

Redundant Fan Walls: Multiple hot-swappable banks of high-speed, high-pressure fans are deployed within the system. These fans are also redundantly configured, so that even if one fan fails, the system can still maintain sufficient airflow for safe operation.

Optimized Heatsinks: All the core chips, and especially the SXM4 GPU modules, are covered by massive passive heatsinks whose fin density and shape have been optimized through computational fluid dynamics simulation to maximize the surface area in contact with the air.

Requirements on the Data Center Environment: This powerful air-cooling system also imposes stringent requirements on the cooling capacity and rack layout of the data center. It must be guaranteed that a continuous supply of low-temperature cold air reaches the fronts of the racks (the cold aisle), and that there is sufficient space behind them to exhaust the hot air (the hot aisle).

4.6.4 The Management Subsystem

BMC (Baseboard Management Controller): The DGX A100 integrates a dedicated BMC chip. It is a miniature computer with its own processor, memory, and network interface, operating independently of the main CPU.

Out-of-Band Management: Through the BMC, administrators can perform complete out-of-band management of the server. Even if the server is powered off, has crashed, or its operating system has hung, as long as the power cable and the management network cable are connected, administrators can remotely:

• Power the server on/off and restart it.
• Monitor the status of all hardware sensors (temperature, voltage, fan speed).
• Access a virtual keyboard, display, and mouse (KVM over IP).
• Mount an ISO image for remote system installation.

This capability is of critical importance for automated operations and rapid fault response in large-scale data centers.

4.7 Chapter Summary

In this chapter, taking the NVIDIA DGX A100 as our blueprint, we completed an in-depth exploration of the design and implementation of a top-tier GPU server. We no longer view a GPU server as a mere accumulation of hardware, but understand it as a complex system, meticulously engineered for a single goal -- maximizing the efficiency of multi-GPU collaborative computing -- in which all subsystems are finely balanced against one another.

Our dissection followed the logic of systems engineering:

We first recognized the value of the DGX, which, by providing a platform that is optimized across the full software-hardware stack, works out of the box, and is backed by enterprise support, greatly lowers the barrier for enterprises to deploy and apply AI, establishing it as the industry's "golden reference design."

At the level of overall design, we saw a clear architecture with the 8-GPU NVLink Fabric as the absolute core, supported by powerful dual CPUs, massive memory, and separated high-speed networks -- every design decision serving the efficient flow of data.

In the CPU and memory subsystem, we understood the importance of a powerful multi-core CPU and vast RAM for data preprocessing, I/O scheduling, and for meeting the challenges of the NUMA architecture. With its abundant PCIe lanes and high core count, the AMD EPYC proved to be the ideal choice.

In the PCIe subsystem, we uncovered the magic of the PCIe switch. It not only expands the number of lanes but, more importantly, builds optimized P2P paths that enable key technologies such as GPUDirect RDMA, opening high-speed direct channels between the GPU on one side and the network and storage on the other.

In the NVLink subsystem, we witnessed the very essence of DGX design. Through 6 NVSwitch chips, it achieves a fully connected topology among the 8 A100 GPUs with 4.8 TB/s of bandwidth, providing an unrivaled hardware foundation for efficient model parallelism (especially tensor parallelism).

Finally, in the auxiliary subsystems of power, cooling, storage, and management, we saw an enterprise-grade pursuit of redundancy, reliability, and maintainability. From the N+N redundant PSUs to the precision-engineered airflow paths and the powerful out-of-band BMC, every detail is designed to safeguard the stable operation of this "computing beast."

Through our in-depth study of the DGX A100, we have acquired a framework for evaluating and understanding the design of any high-end GPU server. Whether we face the next-generation DGX H100 or a competitor from another vendor, we can now see through the marketing to the architectural core, analyzing the strengths and weaknesses of its topology design, its bottlenecks, and its real application scenarios. This is precisely the core capability required of a senior system architect or an AI infrastructure engineer. In the chapters that follow, we will widen our view from a single server to the entire cluster, exploring how to connect hundreds or thousands of such servers to build a true AI supercomputer.