In the first three parts of this book, we systematically and progressively built up the full-stack knowledge required to construct a large-scale GPU computing platform. Like an apprentice, we began by getting to know the most basic "bricks" (GPU chips), learned to build "houses" (GPU servers), and then planned the entire city's transportation (network), warehousing (storage), municipal services (application platform), and management (operations). We have now mastered all the theoretical knowledge, design principles, and key technologies.
Now it is time to pull all of this knowledge together and throw ourselves into a full-scale "combat exercise." The value of theory is ultimately realized in practice. This chapter, "A Practical Case Study of a GPU Computing Platform Serving Machine Learning," serves as the culmination of this book. Here we will step into the role of a chief AI infrastructure architect and, starting from zero, design and implement a complete, end-to-end GPU computing platform for one specific, real, and highly demanding machine learning application scenario. Note that this is a synthetic case (a design exercise) for consolidating the methodology: the scale, budget, and compute figures are stipulated values, not the project record of any real company.
We will take autonomous driving model training, one of the most complex and capital-intensive applications in AI today, as our source of requirements. Autonomous driving makes demands on compute, data, networking, and storage that are comprehensive, multi-dimensional, and exacting, which makes it the perfect "touchstone" for testing our infrastructure design capabilities.
We will follow a complete system design process: beginning with requirements analysis, moving to overall architecture design, and then to the detailed design and implementation of each subsystem, including compute, storage, and networking.
- Requirements Analysis: We will first go deep into the workflow of autonomous driving model training to understand its unique data characteristics, model scale, training paradigm, and the extreme demands it places on infrastructure.
- Overall Design: Based on the requirements analysis, we will establish the platform's core design philosophy, Cloud-Native HPC, which aims to perfectly combine HPC's ultimate performance with the cloud's elasticity and agility.
- Subsystem Design and Implementation: We will then apply all the knowledge learned in earlier chapters to this case, working through requirements analysis, technology selection, and architecture design for the three core subsystems, one by one.
- Compute: How do we choose GPUs and servers? How do we design cluster scale and a hybrid parallel strategy?
- Storage: How do we build a layered storage system to manage PB-level massive data? How do we resolve the bottleneck of high-concurrency data reads?
- Networking: How do we design a multi-plane network to carry the different traffic of training, storage, and management?
- Platform and Operations Integration: While designing the three major subsystems, we will weave in the application platform and operations concepts learned in Chapters 11 and 12, ensuring that what we build is not merely a "bare" HPC cluster, but a modern AI PaaS platform that is easy to use and easy to manage.
Through this complete case exercise, from requirements to implementation, we will link all the previously scattered knowledge points into an organic whole and truly experience the full process of end-to-end system design as a chief architect. This is the final test of our entire learning journey, and the last leap in transforming theoretical knowledge into practical competence. By the end of this chapter, you will not merely "know" how to build a GPU platform; you will "understand" how to think, weigh trade-offs, and create a genuinely excellent solution to the complex, real-world demands you face.
13.1 Requirements Source: Autonomous Driving Model Training
Autonomous driving, and especially high-level L4/L5 autonomy, is hailed as the "jewel in the crown of AI." Behind it lies a machine learning closed-loop system that is data-driven and unprecedentedly massive in scale. To understand how to design infrastructure for it, we must first go to the heart of its workflow.
13.1.1 The "Data Closed Loop" Workflow of Autonomous Driving
The evolution of an autonomous driving system depends on a "data closed loop" that cycles continuously and optimizes itself:
Data Acquisition
A huge fleet of autonomous driving test vehicles collects data on roads around the world, 24 hours a day.
Each vehicle is equipped with a variety of sensors: high-resolution cameras (front, rear, side, and surround views), high-precision LiDAR, millimeter-wave radar (RADAR), IMU (Inertial Measurement Unit), GPS, and more.
These sensors generate enormous volumes of data every second. By the accounts public in the industry, a single test vehicle equipped with multiple cameras and LiDAR can produce on the order of a terabyte of data per hour (the exact figure depends on sensor configuration and sampling rate). A fleet of several hundred vehicles can reach PB-level data generation every day.
Data Ingestion & Storage
The collected data is transmitted back to the data center over mobile networks, or via high-speed wired networks once the vehicle returns to base.
This raw, unstructured data (which we call "Raw Data") needs to be stored reliably and at low cost. Its total volume will reach tens or even hundreds of petabytes, making it a classic "data lake" scenario.
Data Processing & Labeling
Raw data cannot be used for training directly. Algorithm engineers must mine the vast corpus of "raw material" for valuable "corner cases," such as a rare pedestrian crossing, a shift in lighting under harsh weather, or a complex unprotected left turn. This process is called data mining.
The valuable data segments that are mined are then sent for manual or automated labeling. For example, engineers may box out all vehicles, pedestrians, and traffic signs in images and attach labels to them, or perform semantic segmentation on every point in a LiDAR point cloud. The labeled data (which we call "Ground Truth") is the "standard answer" that the model learns from.
Model Training
This is the stage that concentrates computing demand most heavily. Algorithm engineers use the labeled datasets to train various types of deep learning models:
Perception models: such as image-based 2D/3D object detection (YOLO, CenterPoint), semantic segmentation and object detection on LiDAR point clouds, and multi-sensor fusion (BEV-Fusion).
Prediction models: predicting the trajectories of other traffic participants (vehicles, pedestrians) over the next few seconds.
Planning & Control models: deciding the ego vehicle's optimal path and speed based on an understanding of the environment and forecasts of the future.
These models, especially end-to-end large models and BEV (bird's-eye view) models, have enormous parameter counts, and their training must run on large-scale GPU clusters using complex hybrid parallel strategies.
Simulation & Evaluation
A newly trained model cannot be tested on the road directly. It must first be validated on a large-scale, realistic simulation platform.
The simulation platform replays a wide range of real or synthetic traffic scenes to test how the new model performs under various corner cases. A complete regression test may require running millions of simulation cases in parallel across thousands of CPU or GPU nodes.
Deployment & Hardware-in-the-Loop (HIL) Validation
Models that pass simulation validation are deployed to Hardware-in-the-Loop (HIL) test benches, or even real test vehicles, for small-scale road testing.
During road testing, the system continuously collects the scenarios where the new model performs poorly. This fresh "corner case" data is fed back to the data center, where it enters the next iteration of the "data closed loop."
13.1.2 Extreme Infrastructure Demands of Autonomous Driving
From the workflow analysis above, we can distill several core demands that autonomous driving places on infrastructure:
Massive, Layered Storage Requirements
A PB-to-EB "data lake": an extremely low-cost, almost infinitely scalable storage system is needed to archive the vast volume of raw road-collection data. This points to distributed object storage.
A high-performance "training data warehouse": the labeled datasets used for training need to be served by a storage system capable of supporting thousands of nodes reading at high concurrency, low latency, and high throughput. This points to a distributed parallel file system.
Fast checkpoint storage: the model checkpoints produced during training can reach hundreds of GB and must be written quickly, to reduce the training interruption caused by preemption or failures.
Ultra-Large-Scale Heterogeneous Computing Requirements
A large-scale GPU training cluster: model training is the core workload. This requires a computing cluster with thousands of top-tier GPU cards interconnected over a high-speed network, capable of sustaining large-scale distributed training tasks that run for weeks or even months.
A large-scale CPU/GPU simulation cluster: simulation testing places an equally enormous load on compute, and typically consists of a large number of short, parallelizable, relatively independent tasks that need both CPUs and GPUs.
A data processing and analysis cluster: data mining, cleaning, and other ETL (extract, transform, load) tasks usually run on big data clusters built on Spark or Flink.
Extreme-Performance Network Requirements
A high-bandwidth, lossless compute network: this carries the gradient and parameter exchange of large-scale distributed training. The network's performance directly determines how efficiently the cluster can scale.
A high-throughput storage network: this must allow thousands of nodes to pull data from the storage system simultaneously at high speed, so that "data feeding" never becomes the bottleneck.
A high-bandwidth data ingress: high-speed data upload channels are needed from the outside world (the road-collection fleet, labeling centers) into the data center.
Requirements for Elasticity and Agility
Diverse concurrent workloads: the cluster must simultaneously support long-duration training tasks, vast numbers of short simulation tasks, interactive data analysis, and more.
Resource preemption and scheduling: different tasks contend for resources. The platform must support preemptive scheduling (for example, a high-priority pre-release regression test can preempt an ordinary exploratory training task) as well as checkpointing and resumption of training.
Rapid iteration: algorithm engineers need to stand up experimental environments, submit training jobs, and get results quickly. The platform's ease of use and agility are paramount.
These complex and exacting requirements tell us that we cannot simply copy traditional HPC or internet architectures. We need a new architecture that fuses the strengths of both.
13.2 Overall Design: Cloud-Native HPC
Faced with the complex scenario of autonomous driving model training, our overall design philosophy is to build a "Cloud-Native HPC" platform.
High-Performance Computing (HPC): we will draw on the design thinking of traditional supercomputing centers to build the platform's core "engine." This means:
Adopting dedicated, high-density GPU compute nodes.
Adopting a dedicated, lossless, low-latency high-performance compute network (such as InfiniBand or RoCE) to interconnect the compute nodes.
Adopting a dedicated, high-performance parallel file system to support reads of training data.
The goal is to deliver extreme, predictable bare-metal performance for large-scale, tightly coupled distributed training tasks.
Cloud-Native: we will fully embrace the cloud-native technology stack centered on Kubernetes to build the platform's "operating system" and "user interface." This means:
Everything as containers: every application, whether a training task, a simulation task, or a data processing service, is packaged and run inside containers.
Unified resource orchestration: Kubernetes serves as the single, unified resource scheduling and orchestration platform managing all of the cluster's compute, storage, and network resources.
Declarative APIs and automation: through Kubernetes' declarative APIs, we achieve automated infrastructure deployment, elastic scaling of applications, and self-healing.
Agility and elasticity: we leverage the cloud-native ecosystem to give users flexible, on-demand, self-service capabilities for requesting resources and managing environments, improving R&D efficiency.
The essence of Cloud-Native HPC is to run, on one and the same platform, both the "HPC-class" applications that demand extreme performance and low communication latency (like distributed AI training) and the "cloud-class" applications that need elasticity, agility, and isolation (like data analysis, simulation, and online services). It seeks to fuse the "muscles" of HPC with the "brain" of the Cloud.
With this overall design philosophy in place, we turn to the detailed design of each subsystem.
13.3 Compute Requirements Analysis, Design, and Implementation
The compute subsystem is the heart of the entire platform, and its design directly determines the model scale and training efficiency the platform can support.
13.3.1 Requirements Analysis
GPU selection: autonomous driving models, particularly multi-modal fusion models like BEV, have enormous parameter counts and make heavy use of Transformer structures. This requires GPUs with:
Huge memory capacity, to hold the large models, activations, and optimizer states.
Very high mixed-precision compute capability, especially optimized for Transformers.
Very high inter-GPU interconnect bandwidth, to support efficient tensor and pipeline parallelism.
Server selection: we need high-density GPU servers optimized for large-scale AI training.
Cluster scale: this is a key business and cost decision. Suppose that, per the algorithm team's planning, to complete training of a SOTA (State-of-the-Art) model within a reasonable time (say, two weeks), they estimate needing roughly 1024 top-tier GPU cards of sustained compute.
Scheduling requirements: we need batch submission, queuing, and preemption of large-scale distributed training tasks, integrated with the container platform (Kubernetes). At the same time, to raise utilization, we need support for shared GPU scheduling.
13.3.2 Design and Implementation
GPU and Server Selection
GPU: without question, we choose the current flagship NVIDIA H100 80GB SXM5 module. Its 80GB of HBM3 memory, its fourth-generation Tensor Cores with FP8 support, and its Transformer Engine fit the needs of autonomous driving large models perfectly.
Server: we choose the NVIDIA DGX H100, or HGX H100 8-GPU servers built by its certified OEMs (such as Supermicro, Dell, and Inspur) and equipped with eight H100 SXM5 modules. Through NVSwitch, these servers deliver 900 GB/s of fully connected NVLink between the eight cards, making them the ideal platform for efficient tensor parallelism.
Cluster Scale and Physical Layout
Total scale: to satisfy the demand for 1024 cards, we need to deploy 1024 / 8 = 128 HGX H100 8-GPU servers.
Physical unit: we organize these 128 servers into a Scalable Unit (SU). This SU is an independent "supercomputer" with complete compute, network, and storage resources. When we need to expand in the future, we can replicate one or more SUs.
Rack layout: each HGX H100 server draws extremely high power (about 10.2 kW), posing a severe challenge to the rack's power delivery and cooling. We need high-density racks that support liquid cooling or advanced air cooling. Assuming one rack can hold four HGX servers, 128 servers require 32 racks.
Training Strategy and Parallelism
For an ultra-large model running on a 1024-card cluster, we adopt a 3D hybrid parallel strategy:
Tensor Parallelism (TP): we perform tensor parallelism (for example, TP size = 8) across the eight GPUs inside a single HGX server, taking advantage of the 900 GB/s NVLink bandwidth.
Pipeline Parallelism (PP): we split the model's different layers across different servers. For example, we use four servers (32 GPUs) as one pipeline parallel group (PP size = 4, with each stage composed of an 8-GPU TP group).
Data Parallelism (DP): we perform data parallelism across multiple such pipeline parallel groups. In our 1024-card cluster, this yields 1024 / 32 = 32 data parallel replicas (DP size = 32).
At the same time, we use ZeRO-1/2 to optimize how optimizer states and gradients are stored under data parallelism, further saving memory.
Scheduling System Design
Bottom-layer orchestration: we use Kubernetes as the unified resource orchestration layer.
Batch scheduler: the default Kubernetes scheduler alone cannot satisfy HPC-class tasks. We need to introduce a batch scheduler, such as Volcano or Kube-Batch.
Gang scheduling: the batch scheduler supports gang scheduling. A 1024-card distributed training task must have all 1024 of its Pods successfully scheduled at once before it can begin. If even a single Pod lacks resources, the entire task waits, rather than starting some Pods and wasting resources.
Queues and priorities: we support creating distinct task queues (such as a high-priority queue and a normal queue) and support task preemption.
GPU sharing:
For large-scale training tasks, we use the Kubernetes Device Plugin to allocate an exclusive physical GPU to each Pod.
For development, debugging, and small-scale inference tasks, we enable NVIDIA MIG. On some servers, we partition each H100 GPU into multiple GI instances and report them to Kubernetes through MIG's Device Plugin, so users can request them on demand. This greatly improves GPU utilization.
13.4 Storage Requirements Analysis, Design, and Implementation
Storage is what sustains the entire data closed loop, and its design must satisfy the needs for layering, high performance, and concurrency.
13.4.1 Requirements Analysis
Data lake: we need an EB-level, extremely low-cost storage pool to archive the PB-level raw data (Raw Data) produced daily. Writes are continuous, but read frequency is relatively low.
Training data warehouse: we need a high-performance storage system to hold the cleaned and labeled datasets used for model training (roughly PB-level). This system must withstand the high-concurrency read pressure of 1024 GPU nodes and tens of thousands of data loader processes, providing aggregate read bandwidth at the TB/s scale. It must support the POSIX interface and GPUDirect Storage.
Model checkpoints: during training, we need to write hundreds of GB of model checkpoints frequently and quickly.
System and application storage: we need highly available persistent storage for all servers' operating systems, the Kubernetes cluster's etcd, and various middleware (such as databases and Kafka).
13.4.2 Layered Storage Architecture Design and Implementation
We design a classic three-layer storage architecture:
Cold Data Layer (Data Lake): Distributed Object Storage
Technology selection: we use a Ceph cluster built on large-capacity HDDs, enabling its RGW (RADOS Gateway) functionality to provide an S3-compatible object storage interface.
Design points:
Scale: we deploy several hundred storage nodes, each mounting multiple large-capacity (for example, 18TB/20TB) SATA HDDs, to build an EB-level storage pool.
Data protection: we use Erasure Coding (EC) instead of multiple replicas. For example, an EC scheme of k=8, m=3 splits data into 8 data blocks and 3 parity blocks, stored across 11 different OSDs. It needs only 1.375x the storage overhead yet tolerates the failure of any 3 OSDs, saving enormous cost compared with 3x replication (three times the overhead).
Access: the data acquisition and preprocessing cluster writes data into this data lake through the S3 API.
Hot Data Layer (Training Data Warehouse): Distributed Parallel File System
Technology selection: we use the Lustre file system, with all of its backend storage on high-performance NVMe SSDs.
Design points:
Architecture: we deploy a highly available MDS (metadata server) cluster alongside a large-scale data server cluster composed of dozens of OSSs (object storage servers). Each OSS is equipped with multiple high-performance NVMe SSDs.
Network: all Lustre servers (MDS, OSS) and compute nodes (clients) must connect to the same high-performance storage network (which we discuss in the next section).
GPUDirect Storage support: the Lustre client must be configured to support GPUDirect Storage. This lets GPUs bypass CPU memory when reading training data, enabling a data path that goes straight from Lustre OSS to GPU memory.
Capacity and performance: the whole Lustre cluster is sized at the PB level, enough to hold all current training datasets and those of the near future. Its aggregate bandwidth can scale linearly to the TB/s level by adding more OSS nodes, satisfying the "feeding" needs of the 1024-card cluster.
Workflow: before a training task begins, a data preparation job prefetches the subset of data needed for that training run from the cold layer's Ceph object storage into the hot layer's Lustre file system.
Platform and Application Storage Layer: Distributed Block Storage
Technology selection: we reuse the Ceph cluster from object storage, but carve out a dedicated pool composed entirely of SSDs, and enable the RBD (RADOS Block Device) service on this pool.
Design points:
This SSD pool is reserved for applications that need low latency and high IOPS.
Through Kubernetes' Ceph CSI plugin, we can dynamically create and mount RBD-type Persistent Volumes (PVs) for Pods.
Applications:
Provide RBD boot disks for the operating systems of all compute and management nodes.
Provide highly available persistent storage for stateful services such as the Kubernetes etcd cluster, Prometheus, and databases.
Serve as a temporary write target for model checkpoints, exploiting the SSDs' high write performance.
Through this layered storage design, we meet the demanding needs of different scenarios at the most appropriate cost, building a data foundation that is both "unfathomably deep" and "fast as lightning."
13.5 Network Requirements Analysis, Design, and Implementation
The network is the lifeblood connecting compute and storage. For a cluster at the 1024-card scale, network design is the decisive factor between success and failure.
13.5.1 Requirements Analysis
Compute network:
Must support large-scale, high-concurrency RDMA communication among the 1024 GPU nodes.
Must be lossless and low-latency.
Its aggregate bandwidth must be able to carry the dense communication of 3D parallel training.
Must be highly available and scalable.
Storage network:
Must connect all 1024 compute nodes and all Lustre and Ceph storage nodes.
Must provide TB/s-level aggregate bandwidth to meet the needs of concurrent data loading.
Out-of-band management network:
Must connect the management ports of all devices (server BMCs, switches, PDUs, etc.).
Must be physically isolated from all data networks to guarantee the highest level of security and reliability.
13.5.2 Multi-Plane Network Architecture Design and Implementation
We adopt a multi-plane network architecture that is physically separated.
Compute Fabric
Technology selection: InfiniBand (IB). For ultra-large-scale, communication-sensitive HPC-class workloads like our 1024-card cluster, IB, with its native lossless characteristics, hardware-level congestion management, and extreme low latency, is a safer and more performance-predictable choice than RoCE. We use the NVIDIA Quantum-2 400Gb/s InfiniBand platform.
Topology architecture: we adopt a two-layer Fat-Tree (Clos) architecture.
Leaf layer: we deploy a number of Leaf switches. Each HGX H100 server (with eight ConnectX-7 400G IB NICs) connects four of its NICs redundantly to one Leaf switch and the other four to another Leaf switch.
Spine layer: we deploy a Spine layer composed of multiple large Director Switches (such as the NVIDIA QM9700). Every Leaf switch is fully meshed to all Spine switches.
Non-blocking design: we carefully size the number of uplinks between Leaf and Spine to achieve a 1:1 non-blocking network, ensuring that any two GPU nodes can obtain full-rate 400Gbps bandwidth.
SHARP technology: the NVIDIA Quantum-2 platform supports SHARP (Scalable Hierarchical Aggregation and Reduction Protocol), which performs collective communication operations such as All-Reduce directly in the network switches through hardware computation and aggregation, rather than on the endpoint GPUs. This dramatically reduces the latency of All-Reduce and further improves the scaling efficiency of distributed training.
Storage & General Purpose Network
To save cost and simplify cabling, we carry storage and business traffic on a single, physically separated Ethernet network.
Technology selection: we use 400Gb/s Ethernet switches that support RoCE, from NVIDIA Spectrum-X or equivalent vendors.
Topology architecture: we likewise adopt an independent Spine-Leaf Clos architecture.
Connections: all compute nodes, Lustre storage nodes, Ceph storage nodes, and management/API servers connect to this network through their Ethernet NICs.
Logical isolation:
We use VLAN or VRF technology to logically isolate storage traffic from business traffic.
For the storage traffic connecting to Lustre over RDMA, we configure lossless Ethernet (PFC/ECN) on the relevant portions of this network.
For ordinary business traffic and the TCP/IP traffic connecting to Ceph, we run it on standard, lossy Ethernet VLANs.
Out-of-Band Management Network
Technology selection: we use a pair of redundant, simple 10GbE Ethernet switches.
Topology: a simple star or tree topology.
Connections: the BMC ports of all servers, the management ports of all IB and Ethernet switches, PDUs, and so on, are all connected to this completely independent management network.
Through this physically three-plane separated network design, we provide optimal, mutually interference-free channels for each traffic type: the most expensive, best-performing InfiniBand network serves the most precious compute traffic; a high-performance, partitionable Ethernet carries storage and general business traffic; and a simple, reliable management network safeguards the cluster's "lifeline."
13.6 Chapter Summary
In this chapter, we carried out an exhilarating "graduation project" that took us from theory to practice. Taking the highly challenging case of autonomous driving model training as our target, we systematically and end-to-end applied the knowledge system built across all the preceding chapters of this book, and completely designed a world-class GPU computing platform grounded in the cloud-native HPC philosophy.
Our design journey began with a deep insight into the source of requirements. By dissecting the complex "data closed loop" workflow of autonomous driving, we distilled its extreme and varied demands on infrastructure across multiple dimensions: storage, compute, networking, and elasticity. This gave clear direction to every architectural decision that followed.
In the overall design, we established "Cloud-Native HPC" as the core guiding idea, aiming to have it both ways: the ultimate performance of HPC and the agility and elasticity of the Cloud. This determined that our subsequent technology selection would "walk on two legs": using the most hardcore HPC technologies to build the performance engine, and the most advanced cloud-native technologies to build the management and application platform.
In the detailed subsystem design and implementation, we made the following key decisions:
At the compute level, we chose the HGX H100 8-GPU server as the standard compute unit, built out a 1024-card cluster, and designed a 3D hybrid parallel training strategy of TP + PP + DP. For scheduling, we combined Kubernetes with Volcano to support gang scheduling of large-scale batch tasks, and used MIG technology for fine-grained GPU sharing, balancing the needs of large-scale training and day-to-day R&D.
At the storage level, we built a layered, heterogeneous storage system: a low-cost Ceph object store forms the EB-level "data lake"; a high-performance, all-NVMe Lustre parallel file system forms the TB/s-throughput "hot data warehouse," feeding GPUs through GPUDirect Storage; and high-IOPS Ceph block storage provides highly available persistent volumes for systems and applications.
At the network level, we designed a physically three-plane separated network architecture: an extreme-performance 400G InfiniBand network with a Fat-Tree topology carries the core compute traffic, accelerated in hardware by SHARP and other technologies; an independent, high-performance Ethernet carries storage and general business traffic; and a fully isolated out-of-band management network protects the cluster's lifeline.
At this point, we have not only completed a design on paper; more importantly, we have established a complete framework for design thinking. We have learned how to start from a complex business requirement, decompose it layer by layer, make sound trade-offs and selections at every technical point on the strength of our deep knowledge base, and finally integrate all these choices organically into a coherent, efficient, and elegant overall architecture.
This is precisely the core value this book hopes to convey to every reader: not merely a collection of knowledge, but an architectural wisdom. I hope that, through this journey, you are now ready to meet the challenges of the large-model era, and to design and build your own powerful AI infrastructure. The road ahead is vast, and the future is full of promise.