祝融说。

In Chapter 1, we established the essential distinction between intelligent compute (represented by GPUs and NPUs) and general-purpose compute (represented by CPUs). Now, we will aim our microscope at the heart of intelligent compute -- the AI accelerator chip. In this domain, the competitive landscape is characterized by "one superpower and many contenders." NVIDIA, leveraging its first-mover advantage and the formidable CUDA ecosystem, has established a de facto industry standard -- it is the "one superpower." At the same time, domestic alternatives led by Huawei Ascend are rapidly rising. Owing to their strategic value in self-reliance and supply chain security, as well as their notable energy efficiency in specific scenarios, they have become the indispensable "counterweight" in intelligent computing center construction.

For an operations expert at an intelligent computing center, being able to invoke nvidia-smi alone is far from sufficient. You must be like an experienced armorer -- capable not only of identifying the models of different weapons, but also of discerning their internal construction, firepower characteristics, and logistical requirements. This chapter will take you deep into the hardware architectures and software stacks of the two core ecosystems, NVIDIA and Huawei Ascend, and explore how to manage and schedule them in a unified manner within the complexities of the real world. This is not merely a required course in technology selection; it is also the cornerstone for realizing a strategy of domestic technology adoption.

2.1 Decoding the NVIDIA Ecosystem: Ampere/Hopper Architecture, Tensor Core, and NVLink Principles

Whenever AI compute is discussed, NVIDIA is a monument that cannot be bypassed. Since releasing the CUDA platform in 2006, NVIDIA has spent well over a decade transforming the GPU from a gaming graphics accessory into an engine driving the Fourth Industrial Revolution. The secret to its success lies in advancing on two legs at once -- "hardware iteration" and "software ecosystem" -- building a moat of unfathomable depth.

2.1.1 Architectural Evolution: From Ampere to Hopper

NVIDIA's data center GPU architectures are named after physicists, and each generation has delivered a significant leap in performance. We will focus on the two architectures with deep influence on the era of large models: Ampere and Hopper. The specifications cited in this section are public figures for specific products and precisions, provided as order-of-magnitude references only.

Ampere Architecture (Representative Products: A100, A800)

The Ampere architecture was released in 2020, and its flagship product, the A100, was the "first spark" that ignited the large-model arms race. The A100's success stems from several revolutionary designs tailored to AI training workloads:

1. The Third-Generation Tensor Core: This is the heart of the Ampere architecture. Compared with its predecessor, it introduced, for the first time, support for the TF32 (TensorFloat-32) data format. TF32 shares the same dynamic range as FP32 (32-bit single precision, 8-bit exponent) while offering only FP16 (16-bit half precision) precision (10-bit mantissa). This means developers can achieve throughput approaching that of FP16 while maintaining stability close to FP32, with virtually no code changes required -- theoretically, eight times the throughput of FP32. This greatly lowered the barrier to mixed-precision training.
2. Multi-Instance GPU (MIG): The A100 was the first to support partitioning a single physical GPU into up to seven independent GPU instances. Each instance possesses its own dedicated compute, memory, and bandwidth resources, achieving genuine resource isolation. This is especially important for AI inference scenarios, where multiple small models can be deployed safely and efficiently on a single card, dramatically improving GPU utilization.
3. Third-Generation NVLink and NVSwitch: The A100 GPU has 12 NVLink channels, providing up to 600 GB/s of bidirectional bandwidth. Through NVSwitch, a fully connected (full-mesh) internal network can be constructed within an 8-GPU node (such as the DGX A100), enabling any pair of GPUs within the node to communicate at full NVLink bandwidth. This laid the foundation for large-scale distributed training.
4. The Emergence of the A800: It is worth noting that the A800 is a "compliance version" of the A100. Its principal change is the reduction of NVLink interconnect bandwidth from 600 GB/s to 400 GB/s, while its other compute and memory specifications remain essentially consistent with those of the A100.

Hopper Architecture (Representative Products: H100, H800)

The Hopper architecture, released in 2022, was designed specifically for Transformer models and large models at the trillion-parameter scale. It can rightly be described as a "nuclear-grade" presence.

1. Fourth-Generation Tensor Core and the Transformer Engine: The core of the Hopper architecture is its new Transformer Engine. It integrates a new generation of Tensor Cores and can intelligently analyze each layer of a Transformer model, automatically deciding whether to switch between FP8 and FP16 precision. The introduction of FP8 (8-bit floating point) once again doubles compute power. When executing Transformer models, Hopper's throughput is several times that of Ampere.
2. Fourth-Generation NVLink and NVSwitch: The H100 raises single-card NVLink bandwidth to 900 GB/s. More importantly, it integrates the NVSwitch chip directly onto the GPU die, making it far more efficient to construct ultra-large-scale clusters (such as NVLink networks of up to 256 GPUs) and greatly reducing cross-node communication latency.
3. DPX Instruction Set: Hopper introduces a new instruction set for accelerating dynamic programming algorithms, which has important applications in gene sequencing, path optimization, and other fields, further broadening the boundaries of GPU application.
4. The Origins of the H800: Much like the A800, the H800 is a "compliance version" of the H100, with its inter-chip interconnect bandwidth restricted to comply with export control requirements. In terms of single-card compute capability, however, it retains the formidable power of the Hopper architecture.

2.1.2 Tensor Core: The "Dedicated Pipeline" for Matrix Operations

If the GPU is an "army of primary-school students," then the Tensor Core is the "gifted class" within that army, tasked specifically with handling Matrix Multiply-Accumulate (MMA) operations, namely D = A * B + C.

Working Principle: A standard ALU (Arithmetic Logic Unit) can execute only one multiplication or addition at a time. A Tensor Core, by contrast, can complete the multiply-accumulate operation of a small matrix (such as 4x4) within a single clock cycle. It is, in essence, a highly optimized, parallel hardware circuit that hard-wires the entire matrix operation flow, functioning like a dedicated assembly line of exceptional efficiency.

Data Precision and Performance: The performance of a Tensor Core is closely tied to the data precision it supports.

• FP16/BF16 (Half Precision): This is the mainstream choice for large-model training today. FP16 markedly reduces memory footprint and communication bandwidth while leveraging the acceleration of the Tensor Core. BF16 (Brain Floating Point) offers the same dynamic range as FP32, giving it an advantage in training stability.
• TF32 (TensorFloat-32): A format introduced with the Ampere architecture, striking a good balance between ease of use and performance.
• FP8 (8-bit Floating Point): The trump card of the Hopper architecture, pushing performance to new heights and especially suited to inference and certain stages of training.
• INT8 (8-bit Integer): Primarily used for inference acceleration. By quantizing an FP32 model to INT8, one can achieve several-fold performance gains and significant reductions in power consumption.

For AI Infra engineers, understanding and driving the algorithm team toward the correct data precision (such as enabling mixed-precision training) is a critical step in squeezing every drop of performance from the GPU and raising MFU.

2.1.3 NVLink and NVSwitch: Building the "Highway Network" for Compute Clusters

Today, with single-card performance approaching physical limits, AI competition has evolved into a competition at the scale of clusters. The core challenge is how to connect tens of thousands of GPUs efficiently. NVIDIA's answer is NVLink and NVSwitch.

The PCIe Bottleneck: Traditional GPU-to-GPU communication relies on the PCIe bus on the motherboard. But PCIe is a shared bus with limited bandwidth (even PCIe 5.0 x16 provides only 128 GB/s of bidirectional bandwidth), and it requires CPU mediation, incurring higher latency. When eight or more GPUs need to exchange gradients simultaneously, PCIe quickly becomes a bottleneck.

NVLink: The Point-to-Point "Private Highway"

NVLink is a high-speed interconnect bus designed specifically for GPU-to-GPU communication, offering far greater bandwidth and lower latency than PCIe. Physically, it is a direct point-to-point connection between GPUs, with data transmission bypassing the CPU entirely -- like a dedicated enclosed skybridge built between two buildings.

NVSwitch: The "High-Speed Interchange" Between GPUs

When the number of GPUs exceeds two, point-to-point connections become prohibitively complex. The NVSwitch chip emerged to solve this problem, acting as a high-speed switch. Within a standard 8-GPU server (such as the HGX platform), multiple NVSwitch chips work together to construct a fully connected (full-mesh) internal network. This means that any GPU card in the server can communicate at full NVLink bandwidth with the other seven cards -- a capability that is critical for collective communication operations such as All-Reduce.

2.1.4 CUDA: The Unbreachable Software "Moat"

If formidable hardware is NVIDIA's sharp blade, then the CUDA ecosystem is its unbreakable shield.

CUDA (Compute Unified Device Architecture): It is not merely a programming language, but a complete parallel computing platform and programming model. It encompasses:

• The Driver Layer: interacts directly with the hardware.
• The API Layer: provides interfaces for C/C++, Fortran, and other languages, enabling developers to write programs (Kernels) that run on GPUs.
• The Runtime Libraries: manage GPU devices, memory, and task execution.

The Ecosystem: Around CUDA, NVIDIA has constructed a vast matrix of software libraries that covers nearly every application scenario:

• cuDNN: A deep neural network library providing highly optimized primitive operators such as convolution, pooling, and activation functions.
• NCCL (NVIDIA Collective Communications Library): A collective communications library implementing core distributed-training operations such as All-Reduce and Broadcast, optimized for NVLink and InfiniBand.
• TensorRT: A high-performance inference engine capable of optimizing, quantizing, and compiling trained models to achieve the lowest latency and highest throughput in production environments.
• Triton Inference Server: An inference serving framework that supports multiple model formats and hardware backends, simplifying model deployment.
• Nsight: A full suite of performance analysis and debugging tools that help developers pinpoint performance bottlenecks.

The maturity and ease of use of this ecosystem mean that the overwhelming majority of AI frameworks (TensorFlow, PyTorch, JAX) and higher-level applications are developed primarily on top of CUDA. For developers, "switching cards" is not simply a matter of swapping hardware; it may well mean abandoning an entire familiar toolchain and decades of accumulated optimization experience. The migration cost is extremely high.

2.2 The Chinese Champion (Key Focus): A Deep Dive into the Huawei Ascend 910B Architecture, the DaVinci Core, and the CANN Software Stack

Against the backdrop of a national strategy of self-reliance and an increasingly complex international trade environment, building a domestic compute system that does not depend on external supply has become a top priority in China's technological development. Within this wave, Huawei's Ascend series of AI processors -- powered by its self-developed DaVinci architecture and a complete software-hardware ecosystem -- is rapidly becoming one of the "mainstay machines" in intelligent computing center construction.

2.2.1 The DaVinci Architecture: A "Three-Dimensional" Compute Core Born for AI

Unlike the GPU, which traces its "general-purpose" lineage back to graphics rendering, Huawei's DaVinci architecture was purpose-built for AI computing from the very outset. Its design philosophy is "extreme energy efficiency and deep optimization of AI operators." At its heart lies the AI Core, a heterogeneous combination of distinct compute units.

We can picture the AI Core as a highly specialized "AI processing workshop" staffed by three teams working in concert:

1. The 3D Cube (Matrix Compute Unit): This is the soul of the DaVinci architecture and the primary source of Ascend's compute power. It is a hardware acceleration unit dedicated to executing large-scale matrix multiplication (MNK). It is called "3D" because it can be viewed as a 16x16x16 cubic compute array, capable of performing a 16x16 matrix multiplied by a 16x16 matrix (with accumulation) in a single clock cycle -- a total of 16*16*16 = 4096 multiply-accumulate operations. This design is exceptionally efficient for models centered on matrix operations, such as Transformers and CNNs. It is the most advanced, fully automated "matrix-machining machine tool" on the workshop floor.
2. The Vector Unit (Vector Compute Unit): Handles non-matrix, element-wise computations, such as vector addition, subtraction, multiplication, and division, as well as activation function calculations (ReLU, Sigmoid, and so on). It is like the "multi-function workbench" in the workshop, taking up the scattered yet indispensable processing tasks.
3. The Scalar Unit (Scalar Compute Unit): Plays the role of the "workshop foreman," responsible for instruction decoding, task scheduling, and program flow control. It is, in effect, a miniature CPU core.

This heterogeneous "3D Cube + Vector + Scalar" design enables the AI Core to process AI tasks with assembly-line efficiency: the Scalar Unit directs, the 3D Cube handles the most time-consuming large-scale matrix operations, and the Vector Unit attends to finishing touches and auxiliary computations.

2.2.2 The Ascend 910B: A Flagship Training Chip on Par with Mainstream Offerings

The Ascend 910B is currently Huawei's flagship AI training chip, an upgrade over the Ascend 910A. It has been substantially optimized in performance, interconnect, and software ecosystem, with public parameters broadly on par with NVIDIA's A100/A800.

Core Specifications (typical values circulating in public channels; Huawei has not published a complete official datasheet, so verify against the actual batch and official documentation):

• Half-Precision Compute (FP16): An order of magnitude of about 320 TFLOPS is widely cited in the industry. (By comparison, the A100 delivers 312 TFLOPS; both refer to dense tensor-core throughput, and the usable figure depends on software-stack optimization.)
• Integer Compute (INT8): An order of magnitude of about 640 TOPS is widely cited in the industry. (By comparison, the A100 delivers 624 TOPS.)
• High-Bandwidth Memory (HBM): 64 GB of HBM2e, with publicly cited bandwidth typically above 2 TB/s.
• On-Chip Interconnect: Integrates Huawei's self-developed HCCS (Huawei Collective Communication System) technology, supporting high-speed interconnect between multiple cards.
• Inter-Board Interconnect: Supports 100G RoCE v2 networking for constructing large-scale clusters.

The Atlas 900 Training Cluster: Built on the Ascend 910B, Huawei has constructed the Atlas 900 training cluster. This cluster connects hundreds or even thousands of 910B nodes over a high-speed RoCE network and, combined with Huawei's HCCL collective communication library, delivers formidable aggregate compute power and excellent linear scaling. It has supported the training of multiple domestically developed large models.

2.2.3 CANN (Compute Architecture for Neural Networks): Ascend's "CUDA"

If the 910B is Ascend's "body," then CANN (Compute Architecture for Neural Networks) is its "soul." CANN is the AI heterogeneous computing architecture Huawei built to counter NVIDIA's CUDA, serving as the bridge between upper-layer deep learning frameworks and the underlying Ascend hardware. Understanding CANN's layered structure is key to troubleshooting performance issues and performing deep optimization on the Ascend platform.

CANN's architecture can be divided, from bottom to top, into several layers:

1. Chip Enablement Layer (Driver): The lowest layer, responsible for driving the Ascend chip and managing hardware resources. This is the layer with which the npu-smi tool interacts.
2. Compute Acceleration Libraries:
   1. HCCL (Huawei Collective Communication Library): Counterpart to NVIDIA's NCCL, providing collective communication primitives (AllReduce, AllGather, etc.) optimized for Ascend hardware and networks. It is the core library for distributed training.
   2. CBLAS (CANN Basic Linear Algebra Subprograms): Counterpart to cuBLAS, providing high-performance matrix and vector computation functions.
3. AscendCL (Ascend Computing Language): This is CANN's core interface layer, the counterpart to the CUDA API. It provides developers with C/C++ APIs for managing devices (aclrtSetDevice), managing memory (aclrtMalloc), managing streams (aclrtCreateStream), and synchronously launching AI tasks. Developers familiar with CUDA programming will find many similarities in its design philosophy.
4. Graph Engine (GE): This is the most intelligent part of CANN. When an upper-layer framework such as PyTorch or MindSpore issues a computation graph, GE takes charge of it and performs a series of optimizations, including:
   1. Operator Fusion: Fusing multiple small operators (such as a convolution plus an activation function) into a single, larger operator that executes more efficiently, reducing kernel launch overhead and memory reads/writes.
   2. Parallel Optimization: Analyzing dependencies within the graph to maximize the parallel execution of independent operators.
   3. Memory Optimization: Reusing memory intelligently to reduce footprint.
5. Operator Compilation and Execution (TBE & AICPU):
   1. TBE (Tensor Boost Engine): A powerful operator development tool. For standard neural-network operators, GE can directly invoke TBE to generate instructions that execute efficiently on the DaVinci AI Core. Should an "obscure" operator exist in a framework that CANN does not yet support, developers can use TBE -- through a Python-like DSL (domain-specific language) -- to define the operator's computation logic, and TBE will automatically compile it into efficient hardware instructions. This is the linchpin of CANN's ecosystem extensibility.
   2. AICPU: For complex logic or host-side operators that are ill-suited to running on the AI Core, CANN schedules them to execute on a CPU.

A Summary of the CANN Workflow:

The approximate end-to-end flow for a PyTorch-for-Ascend task is:

Python code -> PyTorch frontend builds computation graph -> CANN's PyTorch adaptation layer takes over the graph -> GE performs graph optimization -> GE dispatches the operators of the optimized graph: some to TBE for compilation into AI Core instructions, others to AICPU -> AscendCL and the driver layer dispatch the instructions to hardware for execution -> HCCL handles inter-node communication.

For an AI Infra engineer, this implies:

• A performance problem can arise at any link of the chain: Did the graph optimization fail to take effect? Does an operator lack a TBE implementation and thus fall back to the CPU? Is there an HCCL communication bottleneck?
• The core task of ecosystem adaptation is to ensure that every operator of the upper-layer framework finds an efficient execution path within CANN.

2.3 Unified Management of Heterogeneous Compute: How to Support Multiple Chip Types in a Single Cluster

In an ideal world, we would want all our compute to be homogeneous (for instance, entirely H800), since that makes management simplest. In the real world, however -- owing to historical procurement, domestic technology requirements, cost control, and the fact that the optimal hardware differs from one workload to the next -- a single intelligent computing center will often contain both NVIDIA GPUs and Huawei Ascend NPUs, and possibly AI chips from other vendors as well.

How to present a unified resource view of these hardware components that "speak different languages," and how to schedule, monitor, and manage them on a single unified platform (typically Kubernetes), is an important measure of an intelligent computing center's operational maturity.

2.3.1 The Challenge: A Babel of "Dialects"

The fundamental challenge of heterogeneous management is that each type of hardware has its own independent "technical dialect":

• Drivers and Runtimes: NVIDIA requires the NVIDIA Driver and Container Toolkit; Ascend requires the CANN Driver and Ascend Docker Runtime.
• Device Identifiers: Within a container, NVIDIA GPUs typically appear as /dev/nvidiaX; Ascend NPUs appear as /dev/davinciX.
• Environment Variables: NVIDIA applications depend on NVIDIA_VISIBLE_DEVICES; Ascend applications depend on ASCEND_VISIBLE_DEVICES.
• Monitoring Tools: nvidia-smi versus npu-smi.

If this chaos is left unchecked, operations become a nightmare: you would have to maintain separate node pools, separate base images, and separate scheduling strategies for each hardware type, making it impossible to pool resources and allocate them on demand.

2.3.2 The Solution: Abstraction and Unification Through Kubernetes

Through its plugin mechanism and declarative API, Kubernetes offers us a workable solution for smoothing over the underlying hardware differences. The core weapon is the Device Plugin.

How the Device Plugin Works

The Device Plugin is a standard framework provided by K8s that allows third-party vendors to integrate their hardware resources (such as GPUs, NPUs, and FPGAs) into K8s's resource management system.

A Device Plugin is typically a DaemonSet running on every compute node. Its working flow is:

1. Discover: Upon startup, it scans the current node and, by invoking tools such as nvidia-smi or npu-smi, discovers how many available AI accelerator cards the node holds.
2. Register: It communicates with the Kubelet on the node over gRPC and reports the discovered hardware resources to it. For example, NVIDIA's Device Plugin registers a resource named nvidia.com/gpu, while Huawei's Device Plugin registers huawei.com/npu (or a similar name).
3. Allocate: When a Pod's YAML declares a need for nvidia.com/gpu: 1, the Kube-scheduler locates an idle node that holds the resource. Once the Pod is scheduled to that node, the Kubelet invokes the Allocate method of the Device Plugin on that node. The Device Plugin then prepares the environment required by the GPU/NPU (such as setting environment variables) and mounts it into the Pod's container.

2.3.3 In Practice: K8s Practices for Building a Heterogeneous AI Cluster

Below is a set of typical practical steps for managing both NVIDIA GPUs and Huawei NPUs within a K8s cluster:

Node Labeling

First, K8s needs to distinguish between the different types of nodes. You can use the kubectl label node command to attach clear labels to the nodes:

# Label a node equipped with NVIDIA H800
kubectl label node node-01 accelerator=nvidia-h800
# Label a node equipped with Huawei 910B
kubectl label node node-02 accelerator=ascend-910b

Deploying the Respective Device Plugins

Both the official NVIDIA and Huawei Device Plugins need to be deployed in the cluster. They are typically deployed as DaemonSets and use nodeSelector to ensure they run only on nodes equipped with the corresponding hardware.

• The NVIDIA Device Plugin listens for nodes labeled accelerator: nvidia-h800.
• The Ascend Device Plugin listens for nodes labeled accelerator: ascend-910b.

Declaring Resource Requirements in the Pod Spec

Now, users can declaratively request AI compute in their Pod YAML -- just as they would request CPU or memory -- without concerning themselves with the underlying device details.

Example 1: A PyTorch training task Pod requesting one NVIDIA H800

apiVersion: v1
kind: Pod
metadata:
    name: pytorch-train-nvidia
spec:
    nodeSelector:
    accelerator: nvidia-h800 # Schedule to an NVIDIA node
    containers:
    - name: main
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime # Use a CUDA base image
    command: ["python", "train.py"]
    resources:
        limits:
        nvidia.com/gpu: 1 # Request one NVIDIA GPU

Example 2: A MindSpore training task Pod requesting one Huawei 910B

apiVersion: v1
kind: Pod
metadata:
    name: mindspore-train-ascend
spec:
    nodeSelector:
    accelerator: ascend-910b # Schedule to an Ascend node
    containers:
    - name: main
    image: mindspore/mindspore-ascend:2.2.0-cann7.0 # Use a CANN base image
    command: ["python", "train.py"]
    resources:
        limits:
        huawei.com/npu: 1 # Request one Ascend NPU

The Value of Unified Management:

Through this approach, we achieve a unified resource view. For cluster administrators and users:

• Resource Pooling: All NVIDIA GPUs form one logical resource pool, while all Ascend NPUs form another.
• Unified Scheduling: The K8s scheduler can automatically match tasks to hardware based on labels and resource requests.
• Unified View: With kubectl describe node, you can clearly see the total and already-allocated quantities of both nvidia.com/gpu and huawei.com/npu on every node.
• Integrated Monitoring: By leveraging Prometheus Operator and the appropriate exporters, you can collect metrics from both dcgm-exporter and npu-exporter and present them on the same Grafana dashboard, achieving unified monitoring of heterogeneous compute.

By this point, we have gained a deep understanding of the hardware core and software soul of both the NVIDIA and Huawei Ascend ecosystems, and we have mastered the practical method of "seeking common ground while reserving differences, and managing them in a unified fashion" on the modern cloud-native platform Kubernetes. This lays a solid foundation of hardware knowledge for the AI task scheduling, performance optimization, and fault troubleshooting you will encounter in the chapters ahead.