祝融说。

Welcome to the era of intelligent computing.

At the end of 2022, with the explosive arrival of ChatGPT, a technological revolution driven by large language models (LLMs) swept across the globe. It not only transformed how we interact with information but also profoundly reshaped the underlying logic of the computing industry. Supporting this revolution is no longer the familiar traditional data center (IDC) centered on general-purpose computing, but a new kind of infrastructure built for massive intelligent workloads — the Artificial Intelligence Data Center (AIDC).

For IT operations and SRE engineers accustomed to managing x86 servers, virtualization platforms, and network switches, this presents both an unprecedented challenge and a golden opportunity. Traditional monitoring metrics such as CPU utilization, memory usage, and disk I/O prove inadequate and even misleading in an AIDC that harnesses hundreds or thousands of GPUs and NPUs for training runs lasting months. The DevOps system once built around applications must also evolve into an LLMOps system centered on the model lifecycle.

This chapter serves as your first step into the world of intelligent computing. Starting from the most fundamental differences among compute units, we explore the essential distinctions between an AIDC and a traditional data center. Next, we establish a new set of core metrics that genuinely measure intelligent computing efficiency. Finally, we analyze the "two-tier AIDC" architecture, painting a complete blueprint for intelligent computing center operations. Master these and you hold the "first principles" needed to decode every technical detail that follows.

1.1 From General Compute to Intelligent Compute: The Essential Differences Among CPU, GPU, and NPU

To understand the intelligent computing center, we must first understand the changing form of its most essential production material — compute power. Traditional data centers are the domain of "general-purpose compute," ruled by the central processing unit (CPU). In the AIDC, however, the spotlight belongs to the graphics processing unit (GPU) and the neural processing unit (NPU), which together form the core of "intelligent compute."

1.1.1 CPU: The "University Professor" Skilled in Complex Logic

The CPU, as the "brain" of a computer system, embodies a design philosophy of "extreme generality and powerful logic control." We can liken it to a knowledgeable, rigorously logical "university professor."

Architecture characteristics:

• Complex control unit: The CPU contains extremely intricate control logic for instruction decoding, branch prediction, out-of-order execution, and more, ensuring that programs run efficiently through complex logical flows. This is like a professor who can understand and execute a research plan riddled with "if-then-else" logic.
• Large cache: The CPU is equipped with multiple levels of high-speed cache (L1, L2, L3) for storing frequently used data and instructions, minimizing the latency of accessing main memory (DRAM). This is akin to a professor with a photographic memory and a desk piled with reference books, able to summon needed knowledge at a moment's notice.
• Fewer compute cores (Core/ALU): Relative to its massive control and cache sections, the CPU has comparatively few arithmetic logic units (ALUs). Even top-tier server CPUs typically house only a few dozen to a little over a hundred physical cores. This means the professor, while capable of solving the hardest problems, can handle only a limited number of them at any one time.

Work mode: serial and low-latency

CPUs excel at "serial" tasks or those laced with complex logical branches, such as operating system scheduling, database transaction processing, and web service request handling. These tasks share a common trait: "what to do next depends on the result of the previous step." They demand fast single-core response and intricate logical judgment. The CPU's design goal is to "minimize latency" and complete individual tasks as quickly as possible.

1.1.2 GPU: The "Army of Elementary School Students" Skilled in Parallel Computing

GPUs were originally built for graphics rendering, which exhibits a natural "data-parallel" character — the millions of pixels on a screen can each be independently and simultaneously computed for color. Researchers later discovered that this parallel computing capability applies equally well to scientific computing and deep learning, giving rise to the concept of GPGPU (General-Purpose computing on Graphics Processing Units).

We can liken a GPU to an army of tens of thousands of "elementary school students."

Architecture characteristics:

• Massive number of compute units (CUDA cores/stream processors): A high-end GPU (e.g., the NVIDIA H800) contains tens of thousands of compute units. These units are relatively simple in structure, lacking the complex control logic of a CPU core. Each elementary student can only perform simple arithmetic, but there is strength in numbers.
• Simplified control units and smaller cache: GPUs devote more chip area to compute units rather than to control and cache. The entire "army of elementary students" is commanded by a handful of "drill instructors" (Streaming Multiprocessors, SMs) that synchronously execute the same instructions.
• High bandwidth memory (HBM): To feed tens of thousands of hungering compute units, GPUs are equipped with dedicated High Bandwidth Memory (HBM), whose bandwidth is several times, or even dozens of times, that of traditional server memory (DDR). This is like outfitting the army of elementary students with an ultra-fast logistics supply line, ensuring a steady stream of workbooks (data).

Work mode: parallel and high-throughput

The power of the GPU lies in handling "data-parallel" and compute-intensive tasks, with a design goal of "maximizing throughput." The matrix multiplication at the heart of deep learning is the quintessential example. Multiplying an [M, K] matrix by a [K, N] matrix yields an [M, N] result matrix. This computation can be decomposed into M*N independent dot products, each of which can be handed to a GPU core.

Imagine asking a university professor (CPU) to compute a 4096x4096 matrix multiplication. They would have to work step by step through more than 67 million multiplications and additions, taking an extraordinarily long time. Hand the same task to a GPU with 16,384 cores, and it can theoretically split the work into 16,384 pieces and compute them all at once — the difference in efficiency is astronomical. This is precisely why large model training (whose core is massive matrix operations) must rely on GPUs.

1.1.3 NPU: The "Specialized Calculator" Born for AI

If the GPU is a general-purpose computing tool successfully applied to AI, then the NPU (or AI accelerator) is a chip "born for AI." Its design philosophy is "hardening the core AI operators into hardware to pursue extreme energy efficiency."

We can think of an NPU as a "specialized calculator" custom-built for matrix multiplication and convolution operations.

Architecture characteristics:

• Dedicated compute units: NPUs integrate hardware circuits designed for specific AI algorithms. For example, Huawei Ascend's DaVinci Architecture centers on the AI Core, which embeds massive 3D Cube (matrix compute) units dedicated to efficiently executing matrix multiplication. Google's TPU (Tensor Processing Unit), by contrast, adopts a systolic array architecture, in which data flows like blood through vessels, completing matrix operations efficiently across an array of compute units.
• Higher energy efficiency (Perf/Watt): Because they are dedicated hardware, NPUs consume less power and achieve higher efficiency when executing the AI operators they are designed for. This "specialized calculator" does only one thing, but it does it faster and more power-efficiently than anyone else.
• Relatively lower versatility: The NPU's strength is also its weakness. Its support for non-optimized or novel AI operators may be less flexible than a GPU's. Should algorithms undergo a disruptive change, the NPU may require a hardware redesign.

Work mode: hardware-accelerated operators

When a deep learning framework (such as PyTorch or MindSpore) runs on an NPU, the key operators in its computation graph — matrix multiplication, convolution, and the like — are recognized by the NPU's driver layer (e.g., Huawei's CANN) and dispatched directly to dedicated hardware units (e.g., the 3D Cube) for execution, delivering higher acceleration and lower power consumption than a GPU.

1.1.4 Summary and Paradigm Shift

The essence of the paradigm shift: The transition from a traditional IDC to an AIDC is fundamentally driven by a radical change in the compute workload.

• Traditional IDC: Primarily handles I/O-intensive and logic-intensive tasks. Operations focus on managing CPU, memory, and network bandwidth, monitoring application QPS (queries per second) and RT (response time).
• AIDC: Primarily handles compute-intensive tasks. Operations focus on managing GPU/NPU compute resources, monitoring model training/inference efficiency, compute utilization, and communication overhead.

Understanding the essential differences and the collaborative relationships among CPUs, GPUs, and NPUs is the "first lesson" every intelligent computing center operations engineer must master. It determines whether you can make sound technical judgments when it comes to hardware selection, resource scheduling, and performance optimization in the work ahead.

1.2 Core Metrics for AIDC Operations: HFU, MFU, and Linear Scalability

In traditional IDC operations, we are accustomed to watching CPU utilization. If a server's CPU utilization stays below 10% for an extended period, we consider the resource wasted; if it hovers above 90%, we suspect a bottleneck. Applied mechanically to expensive GPU/NPU clusters, however, this logic leads to gross misjudgments and resource waste.

A single NVIDIA H800 GPU can cost hundreds of thousands of RMB, and a cluster of a thousand H800s represents astronomical hardware and electricity costs. At this point, gauging "whether resources are being used effectively" becomes more critical than ever. To that end, we need a set of more refined operational metrics engineered specifically for the AIDC: Hardware Facility Utilization (HFU), Model FLOPs Utilization (MFU), and Linear Scalability.

1.2.1 Hardware Facility Utilization (HFU)

Definition: HFU, sometimes simply called GPU/NPU utilization, is the proportion of time a hardware accelerator spends in a "non-idle" state over a given period. In plain terms, it is the share of time the "chip's clock is running."

How to observe: This is the easiest metric to obtain.

• On NVIDIA GPUs, check the GPU-Util field from the nvidia-smi command.
• On Huawei Ascend NPUs, check the Utilization field from the npu-smi info command (pay attention to the occupancy of the AI Core and AI CPU).
• In a Prometheus monitoring stack, these low-level metrics can be collected via dcgm-exporter or npu-exporter.

Metric interpretation and pitfalls:

A seemingly perfect GPU-Util: 100% may, in the eyes of a seasoned AIDC operator, mask significant problems. High HFU only means the GPU is "busy" — but busy doing what? It might be efficiently executing matrix multiplication, or it might be "spinning idly." For example:

1. Waiting for data: Data must be read from disk to memory, then copied from CPU memory to GPU memory (a process called H2D Copy). If this chain has a bottleneck (e.g., slow disk I/O, or complex data preprocessing saturating the CPU), the GPU waits for data to compute while its clock may still be running, manifesting as high HFU.
2. Inefficient compute kernel: A very small compute task was submitted (e.g., a Batch Size that is too small). Although the GPU is occupied, it is nowhere near leveraging its massive parallel computing advantage, leaving a great many compute units idle.
3. Frequent synchronization waits: In distributed training, if communication or computation is unevenly loaded across nodes, the faster nodes must wait for the slower ones after finishing. This waiting time also leaves the GPU "spinning."

Conclusion: HFU is a necessary but not sufficient condition. Low HFU definitely signals wasted resources, but high HFU does not mean high efficiency. HFU is the starting point of troubleshooting, not the endpoint. When a training task is underperforming and you first find that HFU is already low, the problem most likely lies on the data side (I/O, preprocessing) or the CPU side.

1.2.2 Model FLOPs Utilization (MFU)

MFU is the "golden metric" of intelligent computing efficiency. It truly answers that soul-searching question: "How much of the theoretical performance of this expensive card am I actually using?"

Definition: MFU is the ratio between the floating-point operations per second actually achieved by an AI model during training or inference (Achieved FLOPS) and the hardware's theoretical peak floating-point operations per second (Peak FLOPS).

MFU = Achieved TFLOPS / Peak TFLOPS (TFLOPS: Trillions of Floating Point Operations Per Second)

An analogy:

A Ferrari's theoretical top speed is 350 km/h (Peak TFLOPS). But because of poor road conditions (memory bandwidth bottleneck), a mediocre driver (insufficient code optimization), or frequent traffic jams (communication overhead), its actual average speed is just 70 km/h (Achieved TFLOPS). This Ferrari's "performance utilization" (MFU) is 70 / 350 = 20%. That means 80% of the money was wasted.

How to calculate and observe:

Computing MFU is relatively involved, requiring information about both the model and the hardware:

1. Peak TFLOPS: This is a hardware parameter available from the chip vendor's official documentation. Note that peak performance typically differs by data precision — FP32, FP16/BF16, and INT8 peak figures vary enormously; some figures are also quoted "with sparsity," with the dense figure roughly half. For instance, per NVIDIA's published specifications for the H100/H800, peak FP8 performance with the Transformer Engine approaches 4000 TFLOPS with sparsity (about half that dense).
2. Achieved TFLOPS: This must be measured with profiling tools.
   1. Theoretical estimation: Achieved TFLOPS = (Model FLOPs per Forward + Backward Pass * Batch Size) / (Single Step Training Time). A model's FLOPs can be estimated with tools such as fvcore. A rough rule of thumb is that, for Transformer models, training FLOPs are approximately 6 * Parameters * Training Tokens.
   2. Direct measurement: Use professional performance analysis tools such as NVIDIA's Nsight Systems or PyTorch Profiler, which can directly analyze the execution of GPU kernels and report the compute power actually achieved.

Metric interpretation and optimization directions:

In publicly reported training systems and papers, an MFU of around 30% is generally considered acceptable for dense-Transformer large model training, while exceeding 50% is regarded as quite excellent (this varies considerably across model architectures, precisions, and parallelism strategies; treat it as an order-of-magnitude reference rather than a hard standard). Low MFU typically points to deeper bottlenecks:

1. The memory wall: Compute units process data too quickly, but the rate at which data can be moved from memory to the compute units cannot keep up, leaving the compute units "starving." This is the most common bottleneck.
2. Insufficient kernel optimization: The compute tasks (kernels) submitted to the GPU do not fully exploit hardware features, such as failing to use Tensor Cores (NVIDIA's units dedicated to matrix operations).
3. Data precision mismatch: On hardware that supports FP16/BF16 mixed-precision training, still computing in FP32 fails to leverage the hardware's acceleration advantage.

Conclusion: MFU is the ultimate yardstick for the "cost-effectiveness" of an intelligent computing task. Raising MFU is the core value of an AI Infra engineer, as it directly affects training cost and turnaround time. When HFU is high but MFU is low, the problem most likely lies inside the GPU (memory bandwidth, compute kernel efficiency) or in the algorithmic implementation.

1.2.3 Linear Scalability

Once single-card performance has been optimized to the limit, we turn to more cards to solve bigger problems. Linear scalability measures the extent to which "many hands make light work."

Definition: Speedup is the performance improvement factor gained by using N compute units (e.g., GPUs) compared with using 1 compute unit.

Speedup(N) = T1 / TN (T1 is the time with 1 GPU, TN the time with N GPUs)

Linear scalability measures how close this speedup comes to the ideal case (N times).

Linear Scalability(N) = Speedup(N) / N

Ideal vs. reality:

• Ideal case: Using 8 cards shortens training time to 1/8 of the single-card time — a speedup of 8 and a linear scalability of 100%.
• Real case (example figures): Using 8 cards, owing to the overhead of cross-card communication, training time may only shorten to 1/6 of the single-card time — a speedup of 6 and a linear scalability of 6 / 8 = 75%. That 25% performance loss is the "distributed overhead." Actual figures depend on the model, parallelism strategy, and network quality.

How to observe:

This is measured experimentally. Keeping hyperparameters such as the Global Batch Size constant, run the same training task on 1, 2, 4, 8, ..., N machines, record the training time (or throughput, e.g., samples/sec) for each configuration, and then plot the "Speedup vs. Number of Nodes" curve.

Metric interpretation and bottleneck analysis:

The shape of the speedup curve is the "ECG" for diagnosing distributed training performance.

The curve flattens: If adding nodes yields ever-diminishing performance gains, communication overhead has become the dominant bottleneck. This typically points to:

1. Network bottleneck: Insufficient cluster network bandwidth (e.g., using ordinary TCP/IP rather than RDMA), or an unreasonable network topology causing congestion.
2. Communication algorithm issues: The collective communication algorithm in use (e.g., All-Reduce) is inefficient, or the volume of data being communicated is too large.
3. The curse of Amdahl's law: The non-parallelizable (serial) portion of the task has come to dominate.

Conclusion: Linear scalability is the key to a cluster's "overall combat effectiveness." An excellent AIDC needs not only strong single-card performance (high MFU) but also strong cluster performance (high linear scalability). Pursuing high linear scalability is the core objective of network architecture design, topology planning, and distributed strategy selection.

The synergy of the three metrics:

An experienced AIDC operations expert deploys this combination like a doctor diagnosing a patient:

1. Task slow? First check HFU. If HFU is low, examine the data path and the CPU.
2. HFU high but MFU low? Look for internal GPU bottlenecks, such as the memory wall, or advise the algorithm engineer to optimize the code and enable mixed precision.
3. Single-card MFU already optimized, but cluster scalability poor? Check the network! Inspect the RDMA configuration, NCCL/HCCL communication logs, network topology, and traffic.

Master the three core metrics of HFU, MFU, and linear scalability, and you hold the "X-ray glasses" that let you see through the operational efficiency of an intelligent computing center.

1.3 Exclusive Architecture Analysis: The "Two-Tier AIDC" System — The Collaborative Logic of Centralized Training and Edge Inference

As large model technology moves from the laboratory into industrial applications, a common observation emerges: the AI lifecycle exhibits two distinct stages — "massive training during development" and "massive inference during application." These two workloads impose fundamentally different demands on infrastructure. Trying to satisfy both with a one-size-fits-all architecture often leads to inefficiency and high cost.

Accordingly, leading practitioners in the industry have gradually evolved a clearly divided "Two-Tier AIDC" system. This system partitions the intelligent computing center into two tiers — a "Central Training Center" and "Edge Inference Nodes" — which collaborate to form a complete AI production and service loop.

1.3.1 Central Training Center: The "Central Factory" for Large Models

The Central Training Center is the "brain" and "heart" of the entire intelligent computing system. Its sole objective is to "efficiently and stably complete large-scale model training and fine-tuning tasks."

Positioning and role: The R&D and production base for models, akin to a national key laboratory or the final assembly plant of the automotive industry.

Hardware configuration characteristics:

• Compute: Deploys top-tier training chips, such as NVIDIA H800/H100 clusters or Huawei Ascend 910B clusters. Scale is massive, typically ranging from hundreds to tens of thousands of cards, in pursuit of extreme total compute power.
• Network: A high-performance network is its lifeline. It must adopt lossless networking based on RDMA (Remote Direct Memory Access), such as NVIDIA's InfiniBand or Ethernet-based RoCE v2. The network topology typically employs a Fat-Tree or the more advanced Dragonfly+, ensuring low-latency, high-bandwidth, non-blocking communication between any two nodes. The performance of collective communication libraries (e.g., NVIDIA's NCCL, Huawei's HCCL) directly determines the cluster's linear scalability.
• Storage: Must be fitted with high-performance parallel file systems such as Lustre or GPFS (Spectrum Scale). Large model training demands high-throughput reads of massive datasets (on the order of TB or even PB) and frequent writes of model checkpoints to prevent losses from training interruptions. Checkpoint files can reach hundreds of GB, placing extreme demands on the storage system's concurrent write performance.

Core operational metrics and focus areas:

• Primary metrics: MFU and linear scalability. The operations team's core KPI is maximizing compute utilization, shortening training cycles, and reducing the total cost of ownership (TCO) per training run.
• Task profile: Dominated by long-duration, high-concurrency batch jobs. A pre-training run for a hundred-billion parameter model may run continuously for weeks or even months, making stability an overriding requirement.
• Operational challenges: The "long tail" posed by cluster scale (a few nodes out of tens of thousands are always failing), scheduling and fault tolerance for massive jobs (Gang Scheduling), the hunt for silent hardware errors, and the pinpointing of network communication bottlenecks.

1.3.2 Edge Inference Nodes: The "Community Service Stations" for Models

If the center is the factory that produces models, the edge inference nodes are the "4S dealerships" or "community service stations" that bring model capabilities to users. Their core objective is "low-latency, high-concurrency model inference serving."

Positioning and role: The direct providers of model capabilities, oriented toward end users or business applications, widely distributed close to data sources or users.

Hardware configuration characteristics:

• Compute: Uses cost-effective, energy-efficient inference chips such as the NVIDIA L40S, L4, T4, or the Huawei Ascend 310 series. These chips perform admirably in single-precision or integer (INT8) inference, with power and cost far below those of top-tier training cards. Inference node scale is flexible — a VPC in the cloud, an on-premises enterprise server room, or even an edge server embedded on a production line.
• Network: Typically uses standard 10GbE or 25GbE Ethernet. Communication demands between nodes are far lower than in training scenarios; the main pressure comes from bearing the massive flood of user-side requests.
• Storage: Emphasizes low latency. Local NVMe SSDs are typically used to store model files, enabling sub-second model loading and switching. Models themselves reside in GPU memory through a caching mechanism to answer real-time requests.

Core operational metrics and focus areas:

• Primary metrics: TTFT (Time To First Token), TPS (Tokens Per Second), concurrent user count, and the service SLA (Service Level Agreement). The operations team cares about user experience and service stability.
• Task profile: Dominated by high-concurrency, low-latency online services, requiring support for elastic scaling, canary releases, A/B testing, and the like.
• Operational challenges: Guaranteeing high availability, balancing request load, dynamic batching (Continuous Batching) for varying request lengths, model quantization and optimization, and co-hosting multiple models on the same machine (resource isolation and preemption).

1.3.3 Two-Tier Collaboration: The Closed Loop of Data and Models

The essence of the "Two-Tier AIDC" system lies in the collaboration and closed loop between the center and the edge.

1. Model distribution (HQ -> Edge): Base models or industry models trained at the center, after a series of optimizations (distillation, quantization, pruning) and compilation for the inference engine, are packaged into standardized images or files. Through a model registry (e.g., MLflow Model Registry) and a CI/CD pipeline, they are distributed securely and efficiently to edge inference nodes around the world.
2. Data feedback (Edge -> HQ): In the course of serving, edge nodes generate large volumes of real-world data — user query logs, model outputs, and user feedback (likes/dislikes). These high-value "hard cases," once cleaned and labeled, are periodically sent back to the center's data lake.
3. Iterative evolution (HQ): The center's algorithm team uses the actual data flowing back from the edge to perform continuous incremental training or fine-tuning on the base model, producing new versions that perform better and solve real problems more effectively.
4. The closed loop forms: The new model is again pushed to the edge through the distribution pipeline, delivering better service to users and collecting newer, more effective data in return. The cycle repeats, forming a "data-driven model iteration flywheel."

Division of labor in the operations system:

This architecture naturally leads to a division of labor within the operations team.

• Central AI Infra team: Focuses on resource management, scheduling optimization, and hardware operations for the large-scale cluster — the stewards of "heavy capital assets."
• Edge LLMOps team: Focuses on model serving deployment, performance monitoring, and rapid iteration — the ones who realize "business value."

Summary

The "Two-Tier AIDC" system is not a mere physical division but a profound operational philosophy. It acknowledges the essential differences among the stages of the AI lifecycle, achieves separate optimization for the two core workloads of training and inference through specialized division of labor, and, through the closed-loop flow of data and models, builds a self-evolving, efficient intelligent computing ecosystem.

Understand this architecture, and you can scrutinize the planning, design, and operation of an intelligent computing center from a higher vantage point — seeing why different hardware, networks, and software stacks are needed, and how they work together to ultimately turn raw compute power into business value. This is the key step from "operations engineer" toward "AIDC architect."