祝融说。

In the first two chapters, we have understood why GPUs have become the computing cornerstone of the large model era from two dimensions: the demands of AI algorithms and the implementation of the software stack. We know GPUs excel at large-scale parallel computing, and we understand how software like CUDA and PyTorch unleash this capability. However, for an architect or engineer aiming to build a top-tier computing center, staying solely at the software abstraction level is far from sufficient. Just as to build a high-performance race car, you must deeply understand every cylinder, every piston, and every pipeline of its engine.

In this chapter, we will transform into hardware engineers, holding a "microscope" to delve into the silicon-level micro-architecture of the GPU chip, dissecting its internal structure. Our goal is to answer a series of fundamental questions: How does the GPU organize its thousands of computing cores? How is its core design philosophy fundamentally different from the CPU? How does data flow, compute, and store within the GPU? What revolutionary innovations have the latest GPU architectures (such as NVIDIA's Hopper) brought specifically to tackle the challenges of large models?

We will start with the overall design philosophy of the GPU, establishing a macro framework and understanding its core idea of "throughput priority." Next, using NVIDIA's H100 (with its GH100 chip at its core), the pinnacle of current data center GPUs, as our primary dissection subject, we will analyze in detail its key components: the Streaming Multiprocessor (SM), the fourth-generation Tensor Core, the Transformer Engine, the DPX instruction set, and High Bandwidth Memory (HBM). Finally, we will briefly introduce other GPU models based on the same Hopper architecture to form a complete product spectrum understanding.

Through the study of this chapter, you will no longer view the GPU as a mysterious "black box." Instead, you will be able to see, with an architect's eye, the subtleties and trade-offs of its design, understand the true physical meaning behind different technical specifications (such as CUDA core count, Tensor Core performance, memory bandwidth), and ultimately make the wisest, most professional hardware selection decisions based on business needs.

3.1 Overall GPU Design

To understand the micro-architecture of a GPU, one must first firmly grasp its macro design philosophy. CPUs and GPUs are products of two distinctly different evolutionary paths in computer architecture, with fundamental differences in their design goals, resource allocation, and core trade-offs.

3.1.1 CPU vs. GPU: Latency Optimization vs. Throughput Optimization

We can use a vivid analogy to distinguish CPUs and GPUs:

A CPU is like a knowledgeable, multi-talented Swiss Army knife expert. It can independently process complex, diverse, and interdependent tasks (serial tasks) at very high speed. To achieve this, it needs a luxurious office (large cache), complex decision-making processes (control logic like out-of-order execution, branch prediction), and the ability to quickly retrieve files (low-latency memory access). Its goal is to minimize the completion time of a single task (Latency).

A GPU is like a large, disciplined, and specialized construction crew. This crew consists of thousands of workers, each mastering only a few simple skills (like carrying bricks, laying walls). But they can all start work simultaneously, collaborating to complete a huge project that can be decomposed into a large number of repetitive subtasks (parallel task). Management of the team is relatively simple, requiring no complex independent decision-making; the more important thing is to ensure all workers have work and the material supply keeps up. Its goal is to maximize the total amount of work completed per unit time (Throughput).

This difference in design philosophy is directly reflected in the allocation of transistor resources on the chip:

CPU chip: A large number of transistors are used to build complex control logic and large caches. The actual Arithmetic Logic Units (ALUs) only occupy a small part of the chip area. This "heavy control, light computing" resource ratio ensures the CPU's excellent performance in handling complex instruction streams and reducing single-task latency.

GPU chip: The vast majority of transistors are used to build a massive number of ALUs. Control logic and caches are relatively simple and streamlined. This "light control, heavy computing" resource ratio allows the GPU to focus almost all its "firepower" on parallel data processing, achieving astonishing raw computational throughput.

3.1.2 Macro Architecture of a GPU: From GPC to SM

A modern, high-end data center GPU chip (such as NVIDIA's GH100) is a highly modular and hierarchical complex system. We can peel back its structure from macro to micro:

1. Full GPU Chip: This is the top-level physical entity, such as a complete GH100 chip. It contains all the computing, memory, and interconnect resources.
2. Graphics Processing Cluster (GPC): The entire GPU chip is divided into several GPCs. A GPC can be seen as a relatively independent "large compute partition" within the GPU. Each GPC has its own rasterizer and other graphics-related components (not typically used in general-purpose computing), but more importantly, it contains multiple Streaming Multiprocessors (SMs).
3. Texture Processing Cluster (TPC): In NVIDIA's architecture, GPCs are further divided into TPCs. Each TPC typically contains two SMs and a PolyMorph Engine.
4. Streaming Multiprocessor (SM): The SM is the most core and basic computing unit in the GPU architecture. We can think of an SM as a "mini CPU core" within the GPU, but it is itself highly parallel. A GPU's parallel computing capability largely depends on how many SMs it has and the internal design of each SM. All CUDA thread blocks are scheduled to execute on a specific SM. The SM contains all the resources needed to execute instructions: CUDA cores, Tensor Cores, register file, shared memory, L1 cache, etc.
5. Memory Subsystem: Coexisting with the computing units is the massive memory subsystem. It includes HBM (High Bandwidth Memory) located outside the chip and connected via extremely high-bandwidth interfaces, as well as the multi-level cache hierarchy on the chip (L1 Cache, Shared Memory, L2 Cache).
6. Interconnect Subsystem: Interfaces responsible for communication between the GPU and the outside world, as well as between GPUs. This includes the PCIe bus for connecting to the CPU and external devices, and NVLink dedicated for high-speed GPU-to-GPU interconnection.

An analogy: The entire GPU chip is like a large factory. The factory is divided into several GPC workshops. Each workshop has several TPC production lines. The most important part of each production line is the SM, the core workstation. The thread blocks dispatched by CUDA programs are like construction work orders, scheduled to different SM workstations for completion. The SM workstation has various tools (CUDA cores, Tensor Cores) and temporary material shelves (registers, shared memory). All workstations share a large central warehouse (HBM VRAM) and use an efficient internal logistics system (cache hierarchy) and external transportation channels (PCIe, NVLink) to obtain and deliver materials.

3.1.3 SIMT Execution Model: The Mechanism of GPU Parallelism

CPUs execute a MIMD (Multiple Instruction, Multiple Data) model, where each core can independently execute different instructions. To simplify control logic and save chip area, GPUs employ a more efficient parallel execution model -- SIMT (Single Instruction, Multiple Threads).

Warp/Wavefront: The basic scheduling and execution unit of a GPU is not a single thread, but a group of 32 threads called a Warp (NVIDIA terminology) or 64 threads called a Wavefront (AMD terminology). In any given clock cycle, the SM selects an "active" Warp and dispatches a single instruction to all 32 threads in that Warp.

Execution Process: These 32 threads simultaneously execute this same instruction on different compute units within the SM, but each processing its own different data. For example, executing an instruction c[i] = a[i] + b[i], the 32 threads in the Warp will respectively compute c[0]=a[0]+b[0], c[1]=a[1]+b[1], ..., c[31]=a[31]+b[31].

Branch Divergence: The biggest challenge of the SIMT model comes from conditional branches. If threads in a Warp encounter an if-else statement, and based on their respective data, some threads need to enter the if branch while others need to enter the else branch, what happens?

At this point, Warp divergence occurs. The hardware executes the two branches serially. First, it executes the if branch, during which only threads that need to enter that branch are active; other threads are "masked" (do not perform operations). After the if branch completes, the hardware then executes the else branch, during which the threads that executed if are masked, and those needing to enter else become active.

This means that if the execution paths of threads within a Warp are inconsistent, the total execution time is the sum of all branch path execution times, not just the longest one. This can severely degrade SIMT execution efficiency. Therefore, an important optimization principle in writing high-performance CUDA code is to avoid branch divergence within Warps.

Understanding the SIMT model explains how the GPU manages thousands of threads with a simple control unit: by organizing them into Warps of 32 threads for unified scheduling and instruction dispatching, greatly amortizing the cost of control logic.

3.2 Nvidia GH100 Chip Architecture Analysis

The NVIDIA H100 is the flagship product based on the ninth-generation data center GPU architecture -- Hopper -- with the GH100 chip at its core. The H100 is a monster processor designed to accelerate large-scale AI and HPC applications, with several major innovations over the previous generation Ampere architecture (A100). We will use the full-spec GH100 chip as an example for in-depth analysis of its key components.

GH100 Key Specifications (Full Version):

Manufacturing Process: TSMC 4N Custom
Transistor Count: 80 billion
GPC Count: 8
TPC Count: 72 (9 per GPC)
SM Count: 144 (2 per TPC)
HBM Interface: 6 HBM3 or 5 HBM2e controllers, 5120-bit memory bus width
L2 Cache: 60 MB
NVLink: 4th Gen NVLink, total bandwidth 900 GB/s

Note: The commercially available H100 SXM5 version enables 132 SMs, while the PCIe version enables 114 SMs. This is standard practice to ensure yield.

3.2.1 Hopper SM Architecture: The New Generation Computing Heart

Each GH100 SM is a powerful parallel processing engine. Compared to the A100's GA100 SM, it maintains the core structure while featuring significant enhancements. A Hopper SM mainly contains:

• 128 FP32 CUDA Cores: The basic units for executing standard single-precision floating-point operations. Like the Ampere SM, the Hopper SM also supports concurrent execution of FP32 and INT32 operations.
• 4th Generation Tensor Core: This is one of the most important upgrades in the Hopper architecture. Tensor Cores are dedicated hardware units for accelerating matrix multiply-accumulate operations. The 4th generation Tensor Core introduces support for the FP8 (8-bit floating point) data format and offers astonishing computational throughput.
• Transformer Engine: A software-hardware collaborative innovation that works closely with the 4th generation Tensor Core. It can intelligently and dynamically select and switch between FP16 and FP8 precision, greatly accelerating Transformer model training and inference without sacrificing accuracy.
• DPX Instruction Set: A new set of instructions for accelerating dynamic programming algorithms, with important applications in gene sequencing, protein structure prediction, path optimization, etc.
• 256 KB Register File: Each SM has a large pool of registers for threads to store private variables.
• 192 KB Configurable L1/Shared Memory: This high-speed on-chip memory can be flexibly configured. For example, it can be set as 128 KB shared memory + 64 KB L1 cache, or other combinations, to suit different application needs. Shared memory capacity and bandwidth are improved compared to Ampere.
• L0 Instruction Cache: Used to cache instructions, improving instruction fetch efficiency.
• Warp Scheduler: Responsible for selecting active Warps from the thread blocks assigned to this SM for scheduling and execution.

3.2.2 4th Generation Tensor Core and FP8 Format

Since its introduction in the Volta architecture, Tensor Core has been NVIDIA's "killer feature" maintaining its leading position in AI. Hopper's 4th generation Tensor Core pushes this advantage to new heights.

Matrix Multiply-Accumulate (MMA): The core function of Tensor Core is to efficiently execute the matrix multiply-accumulate operation D = A * B + C. A Tensor Core can complete a small matrix multiplication and accumulation in a single clock cycle.

Supported Data Precisions:

• FP64: For traditional HPC double-precision computing.
• TF32 (Tensor Float 32): Introduced with Ampere, offering FP32's dynamic range with FP16's precision, accelerating FP32 operations without code modification.
• FP16/BF16: Common half-precision formats for deep learning training.
• INT8/INT4: For deep learning inference, pursuing extreme performance.
• FP8 (E4M3 / E5M2): Hopper's signature feature. FP8 is an even lower-precision floating-point format than FP16, further reducing data volume and computation. FP8 has two variants: E4M3 (4-bit exponent, 3-bit mantissa) with higher precision, suitable for activations in forward propagation; E5M2 (5-bit exponent, 4-bit mantissa) with wider dynamic range, suitable for gradients in backward propagation.

Performance Gain: With FP8 support, H100's theoretical AI peak performance improves markedly over A100. When FP8 and sparsity features are enabled, the H100 SXM5's peak compute is rated at 4000 TFLOPS (4 PFLOPS), roughly twelve times A100's FP16 peak (312 TFLOPS). Note that this is a ratio across different precision tiers — A100 does not support FP8, so the multiple does not represent measured training speedup at the same precision; actual training throughput is further constrained by model structure, parallelism strategy, and communication bandwidth. See NVIDIA H100 product information.

3.2.3 Transformer Engine: The Tailored "Automatic Transmission" for Large Models

Transformer models have become the foundation architecture for today's large language models and many vision models. Their computation is massive, posing severe challenges to hardware. Hopper's Transformer Engine was born to meet this challenge.

Working Principle: The Transformer Engine is a software-hardware system. It scans tensors in the Transformer network layer by layer, and using NVIDIA's heuristic algorithms and analysis of tensor statistical information, it automatically and dynamically decides for each layer and each tensor whether to use FP16 or FP8 precision for computation and storage.

Intelligent Precision Conversion: In parts requiring high precision (like gradient accumulation), the engine uses FP16 or FP32; in parts tolerant of lower precision (like weights and activations), it automatically converts to FP8 for computation. This conversion process is transparent to the user. Developers need only enable the Transformer Engine without manually managing precision. The hardware handles rapid conversion between FP8 and FP16.

Benefits:

• Performance Improvement: FP8 and Transformer Engine can improve training and inference speed for suitable models and numerical strategies, but gains depend on accuracy requirements, parallelism, batch size, communication, and software versions. NVIDIA's H100 product page gives an "up to about 4x over the previous generation" comparison for a specific GPT-3 175B training scenario. That is an upper-bound vendor benchmark, not a fixed multiplier for every large language model. See the NVIDIA H100 product information.
• Memory Savings: Using FP8 to store weights and activations greatly reduces VRAM usage, allowing larger models or bigger batch sizes on a single card.
• Ease of Use: Developers do not need to be mixed-precision experts to benefit from FP8.

3.2.4 Asynchronous Execution and Next-Gen Multi-Instance GPU (MIG)

Thread Block Clusters: The Hopper architecture introduces the concept of "Thread Block Clusters." A cluster is a group of thread blocks that are physically guaranteed to be scheduled within the same GPC. Thread blocks within a cluster can efficiently exchange data and synchronize through a new mechanism called Distributed Shared Memory (D-SMEM). This allows threads in one thread block to directly access shared memory on other SMs within the cluster via atomic operations (load, store, atomic), without going through global memory, greatly expanding the granularity and efficiency of collaboration between SMs.

Asynchronous Execution Units:

Tensor Memory Accelerator (TMA): H100 introduces TMA, a dedicated hardware unit for asynchronous data copying between global memory and shared memory. In previous architectures, such data movement required dedicated CUDA cores to execute instructions. Now, the main compute threads (CUDA cores or Tensor Cores) can issue a copy task to TMA and immediately return to continue computing, while TMA independently completes the data transfer in the background. This deep overlap of computation and data movement further improves SM utilization.

Asynchronous Transaction Barrier: Used to manage and synchronize threads that depend on completion of asynchronous operations (like TMA copies).

2nd Gen Multi-Instance GPU (MIG):

MIG, introduced in the Ampere architecture, allows a single physical GPU to be safely and hardware-isolatedly partitioned into up to 7 independent GPU instances. Each instance has its own dedicated compute resources (SMs), memory, and bandwidth. This is very useful for cloud service providers, allowing them to rent an expensive GPU to multiple users with smaller tasks, improving resource utilization.

Hopper's 2nd gen MIG provides even stronger security and isolation on this basis. It provides dedicated hardware resources for each MIG instance and supports secure tenant isolation in cloud environments. Additionally, through the Thread Block Cluster feature, collaborative computing capability within a MIG instance is enhanced.

3.2.5 HBM3 and L2 Cache: The Lifeline of Data Supply

No matter how powerful the computing engine, it needs sufficient data supply. Hopper also made major upgrades to the memory subsystem.

HBM3 (High Bandwidth Memory 3): The H100 is the first GPU to use HBM3 memory. Compared to the HBM2e used in A100, HBM3 offers improvements in per-pin data rate and number of channels.

H100 SXM5 version: Equipped with 80 GB of HBM3 memory, total bandwidth up to 3.35 TB/s.

H100 PCIe version: Equipped with 80 GB of HBM2e memory, total bandwidth of 2 TB/s. This terrifying memory bandwidth is a key guarantee that H100's thousands of cores will never "go hungry," and is crucial for memory-intensive large model applications.

L2 Cache: GH100 has 60 MB of L2 cache (GA100 had 40 MB). L2 cache is the last level of on-chip cache shared by all SMs. A larger L2 cache can effectively reduce the number of accesses to external HBM memory, lower memory access latency, and improve effective bandwidth, bringing significant performance improvements to applications with high data reuse.

3.2.6 4th Gen NVLink and NVLink Network

4th Gen NVLink: A single H100 GPU has 18 fourth-generation NVLink channels, each with a unidirectional bandwidth of 25 GB/s and bidirectional bandwidth of 50 GB/s. Total bidirectional bandwidth reaches 900 GB/s, 1.5 times that of A100 (600 GB/s). This provides a solid hardware foundation for efficient tensor and pipeline parallelism among multiple GPUs within a single server.

NVLink Network: This is a groundbreaking innovation. On the HGX H100 8-GPU server motherboard, through a new NVLink Switch chip, an all-to-all NVLink connection between the 8 H100 GPUs is achieved. More importantly, through an external NVLink Switch system (NVSwitch), up to 32 HGX H100 servers (256 GPUs total) can be connected into a huge, single computing domain with a complete NVLink Fabric.

This means that in this 256-GPU cluster, any two GPUs can communicate directly through NVLink, with communication performance far exceeding traditional InfiniBand/RoCE based Ethernet. This makes large-scale tensor and pipeline parallelism across multiple servers possible, greatly simplifying the topology design and communication overhead of ultra-large model distributed training.

3.3 Other Hopper Architecture GPUs

Although the H100 is the flagship of the Hopper family, NVIDIA has also released other products based on the same core architecture to meet different market and application needs.

3.3.1 NVIDIA H100 CNX

The H100 CNX is a product that combines an H100 GPU with an NVIDIA ConnectX-7 SmartNIC on a single PCIe card.

Target Scenario: Mainstream AI and data center servers. In these servers, PCIe bandwidth and CPU involvement often become bottlenecks for network communication.

Core Advantage: The direct PCIe connection between the GPU and the SmartNIC, along with the ConnectX-7's hardware offload engine, allows the GPU to bypass the CPU for direct network communication (GPUDirect RDMA). This significantly reduces network communication latency, frees up CPU resources, and is well-suited for large-scale, I/O-intensive distributed AI training and data analysis tasks.

3.3.2 NVIDIA Grace Hopper Superchip (GH200)

The GH200 is a milestone product in NVIDIA's heterogeneous computing journey. It packages an ARM-based Grace CPU and a Hopper GPU into the same module through an ultra-high-speed on-chip interconnect technology -- NVLink-C2C (Chip-to-Chip).

NVLink-C2C: This is the GH200's secret weapon. It provides up to 900 GB/s bidirectional bandwidth, directly connecting the CPU's memory and the GPU's memory. This is 7 times faster than traditional PCIe 5.0 (128 GB/s).

Unified Coherent Memory: Thanks to NVLink-C2C, the GH200 achieves a single memory address space between the CPU and GPU. The CPU can directly access all of the GPU's HBM memory, and the GPU can directly access the CPU's LPDDR5X memory. This means:

Huge Memory Pool: A single GH200 can provide up to 624 GB of quickly accessible memory for the GPU (Hopper's 80 GB HBM3 + Grace's 544 GB LPDDR5X). This enables handling applications that were previously limited by GPU memory capacity, such as ultra-large graph neural networks, recommendation systems, and vector databases.

Simplified Programming: Developers no longer need to explicitly copy data between CPU and GPU memory. The system hardware automatically handles data migration and coherency, greatly simplifying heterogeneous computing programming.

Target Scenarios: Giant AI inference, large-scale graph computing, high-performance computing (HPC). For applications where the model is too large to fit entirely in HBM memory, or requires frequent, massive data exchange between CPU and GPU, the GH200 offers unprecedented performance and ease of use. Multiple GH200 superchips can also be connected through the NVLink network to build even more powerful computing clusters.

3.4 Chapter Summary

In this chapter, we completed an in-depth "dissection" of the GPU hardware architecture. Through this journey, we fundamentally understood the physical basis for why GPUs have become the core engine of the large model era.

We first established the GPU's overall design philosophy: a massively parallel processor designed for extreme throughput. Its "heavy computing, light control" resource allocation strategy and its mechanism for efficiently managing tens of thousands of threads through the SIMT model set it apart from the CPU's "low-latency" design philosophy, onto a path specialized for parallel computing.

We then focused our lens on the current pinnacle of computing power, the NVIDIA H100 (GH100). We then dissected one by one its key internal components, forming a detailed architectural picture: The Hopper SM is its computing heart, integrating FP32 cores, registers, shared memory, and other basic components.

The 4th generation Tensor Core and its support for the FP8 format are the "nuclear weapon" behind the H100's large jump in rated compute.

The Transformer Engine acts like an intelligent "automatic transmission" driver, releasing the power of FP8 through software-hardware collaboration in a transparent, seamless way, specifically designed to accelerate today's most mainstream large model architectures.

New features like Thread Block Clusters and TMA asynchronous copying further tap into the parallel potential within and between SMs, overlapping computation and data movement to the extreme.

On the data supply side, the HBM3 memory at up to 3.35 TB/s and 60 MB of L2 cache together form a powerful data lifeline, ensuring computing cores never "starve."

On the interconnect side, the 900 GB/s 4th generation NVLink and the revolutionary NVLink Network pave the way for building a 256-GPU fully interconnected cluster, making ultra-large-scale model parallelism a reality.

Finally, we also reviewed derivative products of the Hopper family, such as the H100 CNX and Grace Hopper Superchip (GH200), seeing how GPUs are continuously expanding their application boundaries and solving the memory wall and communication wall problems in heterogeneous computing through deep integration with SmartNICs and high-performance CPUs.

Through the study of this chapter, our understanding of the GPU is no longer limited to the vague concept of "having many cores." Now, when we look at an H100 specification sheet, we can understand the deep architectural design and technical implications behind numbers like "132 SMs," "4 PFLOPS FP8 compute," and "3.35 TB/s bandwidth." This deep, bone-level hardware cognition will be the solid foundation for making correct decisions in future computing center planning, hardware selection, performance optimization, and fault troubleshooting.