Close the book, and reflect. Reading this far, our shared journey is drawing to a close. This is not merely the end of a book, but, I hope, the starting point of a new journey for you. Together, like a patient explorer, we have traversed the vast and deep continent of AI infrastructure. From gazing up at the brilliant stars of the large-model era in Chapter 1, to building with our own hands a "city of computing power" for autonomous driving in Chapter 13, we have walked a complete path from ideal to reality, from abstraction to the concrete.
We have been algorithm engineers, feeling the pulse of matrix multiplication on silicon chips; we have been hardware architects, taking a "scalpel" to the microstructure of the GPU; we have been network planners, designing superhighways for data flows; we have been storage specialists, raising layered memory palaces for vast amounts of information; and finally, we put on the hat of chief cloud-platform architect, fusing all of it into an organic, self-breathing, self-evolving intelligence.
At the end of this journey, I invite you to set aside the complex jargon, the architecture diagrams, and the command lines, and to step back with me to a higher summit. From there, let us look back over the ground we have covered, and then gaze out at that more thrilling horizon, already lit by the dawn of AI. This epilogue is not meant to add to your mental "burden," but to be a moment of intellectual "resonance" and "outlook."
I would like to address three topics:
- Reflection and consolidation: In this book, what "fishing skills" did we truly learn, beyond the "fish" itself? I want to distill the book's dense technical detail into a few core architectural principles and first principles that you can carry with you.
- Reality and challenges: Theory is rich, but the real world is lean. Beyond technology, when building and operating a large-scale AI computing center, what equally important -- or even more daunting -- "non-technical" issues will we face? Cost, energy, supply chain, and above all, the "human" factor.
- The future and outlook: AI technology is racing forward at an unprecedented pace, and the infrastructure beneath it must evolve with it. What will the AI computing centers of the future look like? Where are hardware, software, networking, and storage headed? And in the "post-Moore era," how do we extend the legend of computing power?
The Sediment of Thought -- The "Fishing Skills" We Take With Us
Technology changes by the day. Specific GPU models, software versions, and network protocols will all become obsolete, but the design thinking and architectural principles running through them endure. If this book is a treasure house of knowledge, what I most hope you take away are these master keys that unlock every future technology.
The Law of Bottleneck Migration: The Eternal Art of System Balance
Looking back over the entire process of building an AI platform, you will find a recurring pattern: we solve a bottleneck, yet the bottleneck does not disappear -- it simply migrates to the next link in the system. This is the celebrated "Law of Bottleneck Migration" of systems engineering, and it is the core methodology of our infrastructure architecture design.
From CPU to GPU: At first, we found that the CPU's serial computing capacity was the bottleneck for AI algorithms, so we introduced the GPU. The GPU's powerful parallel computing resolved the compute problem, but data feeding soon became the new bottleneck.
From memory to network/storage: To keep the GPU fed, we equipped it with HBM high-bandwidth memory. Per-card performance rose, but growing model scale demanded distributed training. Communication between GPUs -- the network -- became the new bottleneck.
From network to storage: We built a high-speed compute network with InfiniBand/RoCE, resolving the gradient-synchronization bottleneck. Yet as data parallelism grew, with hundreds or thousands of nodes pulling training data at once, the storage system's concurrent read capacity became the new bottleneck.
From physical to virtual: We addressed the storage bottleneck with parallel file systems, building a powerful physical infrastructure. But as users and tasks multiplied, resource allocation efficiency, utilization, and isolation became new bottlenecks, and so we introduced cloud-native computing, virtualization, and resource scheduling.
This process resembles a game of "Whack-a-Mole": we are forever hunting and striking the mole that pops up. The value of a great architect lies not in designing some single "invincible" technology, but in holding a global view of the system. A great architect can foresee where bottlenecks will emerge and lay groundwork in advance, coordinating the design and balanced configuration of every subsystem -- compute, network, storage, software -- so that the whole "barrel" has no conspicuous short plank.
So when you face a performance problem in the future, do not stop at the point where it occurs. Like a seasoned physician practicing "inspection, auscultation, inquiry, and pulse-taking," trace the data's complete path through the system -- from generation, to transmission, to processing, to consumption -- and find the narrowest "defile." System balance is an architect's "sixth sense" that transcends any particular technology, and it is the first core idea this book hopes to hand you.
Layering and Decoupling: The Infallible Key to Complexity
AI infrastructure is an extraordinarily complex mega-system. Faced with such complexity, the most effective instrument of the human mind is "divide and conquer" -- that is, layering and decoupling.
Layering of the physical and the logical: We split the network into physical and virtual networks (VPC), and storage into physical storage and logical volumes/buckets. The physical layer below pursues raw performance and connectivity; the logical layer above pursues flexibility, security, and multi-tenant isolation. This separation lets us optimize the two layers independently and in parallel.
Separation of functional planes: We divided the cluster network into four distinct planes -- compute, storage, service, and management. Each kind of traffic has its own dedicated channel, free of interference. This is like a city's traffic planning, which separates high-speed rail, freight, local transit, and emergency lanes, greatly improving the stability and manageability of the whole system.
Abstraction levels in the software stack: From CUDA libraries, to PyTorch/TensorFlow frameworks, to the Kubernetes orchestration platform, each layer abstracts and encapsulates the one beneath it. This abstraction hides underlying complexity, letting developers at each level focus on their own domain without fretting over hardware details. The democratization of AI is, to a large degree, a history of "ever-rising abstraction levels."
Decoupling through microservices: At the application layer, we split monolithic applications into microservices. Each service can evolve independently, and teams collaborate through APIs. This decoupling at the organizational level mirrors the decoupling at the technical level, and together they unleash the productivity of the R&D organization.
The essence of layering and decoupling is the definition of clear interfaces and boundaries. In a healthy system, components should behave like "high-cohesion, low-coupling" black boxes. The architect's core craft is to design what these "boxes" do and the "contracts" by which they interact. When you confront a sprawling, chaotic system in the future, pick up the "scalpel" of decoupling first: sort out clear layers and boundaries, and complex problems will often resolve themselves.
Hardware-Software Co-Design: The Only Road to Peak Performance
In the domain of AI infrastructure, pure software optimization or pure hardware stacking can no longer deliver order-of-magnitude leaps. The breakthroughs of the future must come from deep, collaborative hardware-software co-design.
This book is full of such examples:
NVIDIA's Tensor Core and Transformer Engine: This is not a bare hardware unit but a hardware-software system. The hardware (Tensor Core) provides FP8 compute capability, while the software (compiler and runtime libraries) intelligently analyzes the model, dynamically selecting and switching precision, and ultimately delivers enormous performance gains transparently to the user. The GPUDirect family: Whether GPUDirect RDMA or GPUDirect Storage, the core idea is the tight coupling of software (drivers, API libraries) with hardware (GPU, NIC, storage controllers) to open a direct data path that bypasses the CPU.
SHARP in InfiniBand: It offloads the collective-communication algorithm All-Reduce -- originally the province of upper-layer software (NCCL) -- down into the network switch's hardware, achieving extreme acceleration of communication operations.
The DPU/SmartNIC: This is the fullest expression of hardware-software co-design. It hardens infrastructure tasks -- networking, storage, security -- that the CPU's software used to handle, onto programmable dedicated hardware, freeing the CPU entirely to attend to the application.
This trend tells us that the future AI infrastructure engineer can no longer be a specialist in a single field. You must "know software and hardware alike." You must understand the needs of AI models, read PyTorch source, analyze NCCL communication patterns, and at the same time read GPU microarchitecture diagrams, understand PCIe topology, and grasp the principles of RDMA. Cross-stack capability will become the gold standard by which a top AI infrastructure professional is measured.
Automation: The Ultimate Answer to Large-Scale Operations
When a cluster grows from 10 servers to 1,000, operational complexity does not merely multiply by a factor of 100 -- it grows exponentially. At scale, any operation that depends on human hands is fragile, inefficient, and unreliable. Automation is the only answer that allows a large-scale system to run stably.
The automation we discuss in this book is end-to-end and full-stack:
Infrastructure as Code (IaC): Using tools such as Ansible and Terraform, we describe and manage the configuration and deployment of physical servers, networks, and storage as code.
Declarative APIs and the controller pattern: Represented by Kubernetes, we no longer care about "how" (imperative) but only about "what" (declarative). We submit a "wish list" of a desired state, and the system's controllers work tirelessly and automatically to bring reality into line with our wishes.
GitOps: We link application deployment and infrastructure configuration to a Git repository. Any change must pass through a commit and a code review. CI/CD pipelines then apply those changes to production automatically. This brings version control, auditability, and collaboration to our operations.
AIOps (AI for IT Operations): This is a higher-order form of automation. We apply AI and machine learning to analyze vast volumes of monitoring data and logs, performing automatic anomaly detection, root-cause analysis, capacity prediction, and intelligent alerting -- and even achieving partial automatic fault healing.
Automation is not merely a tool; it is a culture and a way of thinking. It demands that we convert every repetitive, manual, experience-driven operation into a standardized, repeatable, code-driven process. It demands that we trust the system, empower the system, and let machines do what machines do best -- freeing human beings from drudgery to pursue the more creative work of architectural design and system optimization.
The Challenges of Reality -- The Beasts Beneath the Iceberg
Building an ideal AI computing center is, to be sure, a formidable technical undertaking. But in the real commercial world, equally large -- and even harder to tame -- "beasts" lurk beneath the iceberg. As architects, we must reckon with them honestly.
The "Black Hole" of Cost: Total War on TCO
The cost of a large-scale GPU cluster is staggering. A premium H100 GPU commands a steep price, and a cluster of 1,024 H100s represents an astronomical hardware procurement bill. Yet this is only the capital expenditure (CAPEX). Even more "frightening" is the relentless operating expenditure (OPEX).
Electricity: This is the largest "cash-devouring beast" in OPEX. The compute equipment of a 1,024-card H100 cluster alone can draw more than 1 MW of power. Add cooling, networking, storage, and other supporting systems, and the data center's total consumption climbs even higher. Over a year, the electricity bill alone can be staggering.
Cooling: Sweeping megawatts of heat out of the machine hall requires formidable refrigeration -- air conditioning, chilled water, liquid-cooling loops -- which themselves consume enormous energy.
Networking: High-speed InfiniBand/Ethernet switches, optical modules, cabling, and cross-region, cross-public-network bandwidth all come at a steep price.
Personnel: Maintaining such a complex system demands a team of outstanding network, storage, systems, and AI-platform engineers, whose salaries are a major investment.
Software licensing: Commercial vGPU software, databases, monitoring systems, and the like can carry expensive license fees.
A modern AI infrastructure architect must therefore also be a cost economist. Every technical decision must survive rigorous TCO (Total Cost of Ownership) analysis.
When choosing a GPU-sharing scheme, MIG offers excellent hardware isolation -- but will cGPU's extreme utilization save enough GPU procurement and electricity costs over the long run to justify it?
When choosing a network, InfiniBand delivers peak performance -- but does the RoCE route offer a better TCO once hardware procurement and operations headcount are counted?
When designing storage, an all-flash parallel file system is exceedingly fast -- but can a far cheaper hybrid storage scheme, buoyed by smarter data prefetching and caching, deliver comparable results?
Raising resource utilization is the most effective lever for cutting TCO. This is precisely the business value at the heart of the GPU virtualization and shared-scheduling technologies discussed in Chapters 7 and 8. As a worked example: if a cluster's average utilization really does rise from 10% to 50%, that quintuples the effective compute capacity without any added hardware — which, for a thousand-GPU cluster, corresponds to replacing purchases on the order of hundreds of millions of yuan. How much can actually be gained depends on workload mix and the quality of the shared-scheduling implementation, and should be verified against cluster statistics.
The "Colossus" of Energy: The Era's Mandate for Green Computing
Bound up with cost is the increasingly urgent issue of energy and the environment. Large models have been called "electricity tigers," and their staggering carbon emissions have drawn widespread public concern; the energy appetite of a single AI supercomputing center rivals that of a small city.
As the builders of AI infrastructure, we bear a serious social responsibility. We must weave the ideal of Green Computing into every link of our design.
PUE (Power Usage Effectiveness): This is the core metric of data center energy efficiency, defined as PUE = total data center energy consumption / IT equipment energy consumption. An ideal PUE is 1.0. We need more advanced cooling (liquid cooling, free air cooling) and more efficient power distribution (high-voltage DC, UPS) to approach that ceiling as closely as possible.
Hardware performance per watt: When selecting hardware, we must look beyond peak performance to performance per watt. This is one reason specialized hardware such as ASICs (like the TPU) and DPUs is becoming ever more important.
Software scheduling optimization: The scheduler can be made more "intelligent" -- for instance, running large training jobs during off-peak rate periods when electricity is cheap; routing temperature-insensitive tasks to the data center's hot spots; and automatically putting some servers to sleep when cluster load is low.
Algorithm and model optimization: Infrastructure teams must work closely with algorithm teams to push toward more energy-efficient model design -- using quantization, pruning, and distillation to train smaller, leaner models, and exploring more frugal training methods such as sparse training.
Clean energy: Locating data centers in regions rich in wind, solar, and hydro power is the ultimate solution.
The competition over AI computing power in the years ahead will not be merely a contest of scale; it will be a contest of energy efficiency. Whoever produces more intelligence with less energy will prevail in the long run.
The "Shackles" of the Supply Chain and the "Desert" of Talent
Supply-chain risk: High-end GPUs, switch chips, and other core hardware currently depend on a handful of elite suppliers. Geopolitics, capacity constraints, pandemics -- "black swan" events of every kind -- can fracture the supply chain. The "nowhere to find a single card" situation has repeated itself over the past few years. This obliges us to build diversity and openness into our architecture. In technology selection, can we bring in AMD or Intel GPUs as backups or in mixed deployments? Can we lean more on white-box switches and open network operating systems? Building a more resilient supply chain that is not "locked in" to a single vendor is a strategic consideration of the first order.
Scarcity of talent: The "Cloud-Native HPC" technology stack this book describes has an unprecedented breadth and depth. People who simultaneously master distributed training, Kubernetes, high-performance networking, distributed storage, and the underlying hardware are as rare as phoenix feathers. Talent shortage is the single biggest bottleneck holding many enterprises back from building their own AI platforms. This forces enterprises to weigh "building in-house" against "going to the cloud" with great discernment, and to invest heavily in internal talent development and knowledge infrastructure.
The Horizon Ahead -- How AI Infrastructure Will Evolve
We stand at a singularity of technological explosion. AI infrastructure is evolving even faster than the AI algorithms it supports. Standing at this moment and gazing at the horizon five to ten years out, we can make out several clear and exciting trends.
Hardware in "Alliance": Heterogeneous Fusion and the Chiplet Revolution
The slowing of Moore's Law means we can no longer rely on the brute-force growth of transistor counts in a single chip to push performance. The computing power of the future will come from smarter architectural innovation and heterogeneous fusion.
XPU convergence: We have already seen the birth of the Grace Hopper superchip, which "glues" a powerful CPU and GPU together over ultra-fast NVLink-C2C to form a unified memory space. That is only the beginning. Before long, CPUs, GPUs, DPUs, FPGAs, and a host of dedicated AI accelerators (ASICs) will be integrated ever more tightly into a single package -- even onto a single die -- forming a true "XPU" heterogeneous computing platform. Applications will dispatch tasks across these different compute units seamlessly and dynamically, according to the nature of their workloads.
The Chiplet revolution: Manufacturing a single large, flawless monolithic die is becoming ever more difficult and costly. Chiplet technology offers a new paradigm: decompose a large chip into many small, functionally independent "chiplets," then interconnect them at high speed with advanced 2.5D/3D packaging, much as one assembles Lego bricks.
Advantages:
Better yield, lower cost: Small dies have far higher manufacturing yields than large ones.
Flexibility and customization: Like ordering a la carte, one can combine chiplets from different process nodes and different vendors -- a 7nm compute chiplet paired with a 14nm I/O chiplet -- to rapidly tailor a chip to a specific need.
The future: The GPU or AI processor of tomorrow may well be a "system on a package" made up of multiple compute chiplets, I/O chiplets, HBM chiplets, and more. AMD's MI300 series has already shown the way.
The "Unbounded" Network: Computing and Networking Grown Into One
The network will no longer be a mere "pipe" connecting servers; it will become part of the fabric of computation itself.
The expansion of NVLink networking: We have seen the 256-GPU NVLink Fabric built on NVSwitch with the H100. In the future, that fabric will scale to thousands or even tens of thousands of nodes, producing a true "Rack as a Computer" architecture. Within that vast unified memory address space, any GPU will reach the memory of any other GPU at low latency, as if it were local. This will overturn the programming paradigm of distributed computing entirely.
In-network computing: We have already seen SHARP offload collective communication into the switch. In the future, more computation -- data filtering, aggregation, even simple AI inference -- may be "handled along the way" by DPUs or programmable switches in the network path. The network will evolve from a passive mover of data into an active processor of data.
"Intelligent" and "Tiered" Storage
Intelligent data management: The storage system of the future will be more than a container for data; it will be an intelligent data platform. It will integrate deeply with the AI workflow engines above it (such as Kubeflow and Flyte), and be able to understand the "semantics" of data.
Automatic tiering: The system will analyze data access patterns on its own, proactively and transparently migrating "hot" data that training jobs are about to consume from cheap object storage up into high-performance parallel file systems or local NVMe caches.
Data-aware scheduling: When scheduling a training job, the cluster scheduler will take data locality into account, placing compute as close as possible to the nodes holding the data it needs -- "computing moves to the data," rather than "data is hauled to the computing."
The rise of in-memory storage: As a new generation of interconnect protocols such as CXL (Compute Express Link) matures, we will be able to link large pools of persistent memory or ordinary DRAM through a fabric, building shared memory storage pools at the terabyte or even petabyte scale. For latency-sensitive applications such as online feature stores and graph databases, this will bring a revolutionary leap in performance.
Software-Defined "Everything" and AI-Native Platforms
Beyond Kubernetes: Kubernetes is king today, but it was designed for stateless microservices. For the stateful, batch-oriented, HPC-class AI workload, its support still has many "gaps." In the future we will see more "AI-native" cloud operating systems. They will treat batch schedulers like Volcano, service meshes like Istio, AI workflow engines like Kubeflow, and deep awareness of heterogeneous hardware and tiered storage as first-class citizens, offering a more unified and efficient programming and management model.
AI for Infrastructure: AIOps will deepen. AI will be used not only for reactive fault discovery, but for proactive, predictive system self-optimization.
An AI scheduler can learn the runtime patterns of every historical job to forecast a new job's resource demands and duration, and so make optimal scheduling decisions.
An AI network controller can analyze global traffic in real time, anticipate congestion before it forms, and adjust routing dynamically to head it off.
An AI storage manager can forecast the "heat" of data and make optimal decisions on caching and prefetching.
The entire infrastructure will become one vast reinforcement-learning agent, whose objective is to maximize resource utilization and minimize total cost of ownership (TCO) while meeting every application's QoS requirements.
Conclusion: Become the "Foundation Builder" of What Is to Come
We stand at an unprecedented confluence in history. On one hand, the emergence of large models has brought us, for the first time, within genuine reach of the possibility of general artificial intelligence; on the other, the end of Moore's Law has placed formidable obstacles in the way of extending the legend of computing power.
This is at once a challenge and an opportunity.
The battlefield of AI infrastructure has grown from the "speed and passion" of chasing raw performance into a "total war" over system balance, energy efficiency, cost economics, and intelligent automation. In the end, victory will go to the "system thinkers" -- those who understand applications most deeply, coordinate software and hardware most deftly, optimize TCO most relentlessly, and embrace openness and fusion with the greatest foresight.
This book has drawn you a map to that future and packed all the gear you will need to cross the continent. But the real journey begins the moment you close the book.
Design, build, optimize. Become the one who lays the final foundation stone for the birth of intelligence.
The horizon ahead is in your hands.
Thank you for reading.