祝融说。

With the rapid advancement of information technology, we find ourselves in an entirely new era driven by data and empowered by intelligence. AI is no longer a distant concept from science fiction; it has permeated every aspect of social production and daily life, becoming the core engine of the Fourth Industrial Revolution. In recent years, large-scale pre-trained models (LLMs) represented by the GPT series and LLaMA have achieved breakthroughs. Their powerful capabilities in natural language understanding, generation, and reasoning, as well as their extension into multimodal domains, herald the dawn of AGI, ushering in a magnificent "era of large models."

Yet, above the iceberg is the model's stunning capability; below it lies the unprecedentedly vast computing, storage, and network infrastructure supporting its training and inference. The "large" in large models is reflected not only in their hundreds of billions or even trillions of parameters but also in the extreme demands they place on infrastructure. The complexity and performance of a model exhibit an almost rigid positive correlation with the computing resources it consumes. Therefore, understanding the special demands that AI, especially large models, places on infrastructure is the logical starting point for building an efficient, stable, and scalable large-model computing center.

This chapter serves as the book's opening, aiming to paint a macroscopic picture for the reader. We will first return to the basics, exploring the connotation and evolution of the concept of "AI." Next, we will get a preliminary glimpse of machine learning algorithms, the core driving force behind modern AI. Then, by dissecting a classic linear regression case, we will intuitively reveal the computational essence of machine learning algorithms. Finally, and most importantly for this chapter, we will systematically analyze the special demands machine learning algorithms place on computer hardware, particularly on different computing units such as CPUs, GPUs, and TPUs/NPUs, clarifying why GPUs have become the main engine of AI computing and why we need to build entirely new infrastructure for the era of large models.

It should be noted that this book is not a neutral hardware procurement manual, but an extension of the RC theoretical system (Process Realism: Observational Convergence and the Generation of Certainty) into the domain of intelligent computing infrastructure: building and governing an intelligent computing center is, in essence, the construction of a multi-scale resource order — organizing chips, servers, networks, storage, scheduling, and operations into a determinate assemblage capable of sustained evolution. The RC theory of order holds that "the capacity to absorb and process stress determines the survival of an order," which corresponds to the capacity planning, bottleneck identification, and redundancy design that recur throughout this and later chapters. Its theory of practice — "there is no optimal decision, only a sustainable decision" — corresponds to the stance running through this book: every choice must state its applicable range and failure conditions, and any "optimal configuration" is only a stage-appropriate solution under a particular workload and point in time. Readers interested in the philosophical foundations may trace back from that entry point.

1.1 What Are We Really Talking About When We Talk About AI?

The term "AI," since its formal introduction at the Dartmouth Conference in 1956, has continuously evolved and enriched in both connotation and scope. For the public, AI is often associated with self-aware robots, the "Skynet" from movies, or omniscient digital assistants. In academia and industry, the definition of AI is more rigorous and pragmatic.

1.1.1 The Historical Evolution and Rival Schools of AI

The history of AI has not been smooth sailing. It has experienced several "AI winters" of inflated expectations and shattered bubbles. We can broadly divide it into several stages:

1. Early Beginnings and Golden Age (1950s - 1970s): The Dartmouth Conference marked AI as an independent discipline. Early researchers were optimistic that machine simulation of all human intelligence was imminent. The mainstream of this period was Symbolism, also known as logicism or GOFAI (Good Old-Fashioned AI). Symbolists believed that the core of intelligence was the representation of knowledge and logical reasoning. They devoted themselves to building complex knowledge bases and expert systems, solving problems through formalized symbol manipulation, such as theorem proving and board games. Early checkers programs and the birth of the LISP language were outstanding achievements of this era.
2. The First AI Winter (1970s - 1980s): As research deepened, people found Symbolism facing huge challenges. The first was the knowledge acquisition bottleneck -- building complete, contradiction-free knowledge bases for the complex real world was nearly impossible. The second was computational complexity -- the search space for many problems exploded exponentially with scale, exceeding the limits of contemporary computers. When promised grand goals failed to materialize, government and corporate funding dried up, and AI research entered its first downturn.
3. The Revival of Connectionism and the Second AI Winter (1980s - 2000s): Running parallel to Symbolism was Connectionism. Inspired by the neural network structure of the human brain, it held that intelligence arose from the collective behavior of a large number of simple, interconnected processing units (neurons), rather than complex symbol manipulation. As early as 1943, McCulloch and Pitts proposed the M-P neuron model. In 1958, Frank Rosenblatt invented the "Perceptron," which can be seen as the prototype of modern neural networks. However, in 1969, Marvin Minsky pointed out in his book Perceptrons that a single-layer perceptron could not solve linearly inseparable problems like XOR, which dealt a severe blow to Connectionist research. Although the rediscovery of the backpropagation algorithm and the proposal of multi-layer perceptrons (MLPs) theoretically solved this, limited computing power and data at the time made training deep networks impractical. Coupled with the bursting of the expert system commercialization bubble, AI entered its second winter in the 1990s.
4. The Age of Deep Learning and Large Models (2010s - Present): Entering the 21st century, two key factors paved the way for AI's resurgence: Big Data and High-Performance Computing (HPC). The proliferation of the internet generated massive data, providing ample "nourishment" for machine learning models. At the same time, parallel computing hardware, represented by GPUs, offered floating-point computing capability perfectly suited to the needs of neural network training.

In 2012, Alex Krizhevsky et al., using the deep convolutional neural network (CNN) AlexNet, won the ImageNet image recognition competition by a landslide, with an error rate far below previous methods. This event is seen as the tipping point of the deep learning revolution, marking the complete victory of Connectionism. Since then, deep learning has advanced rapidly in computer vision, natural language processing, speech recognition, and other fields, spawning landmark models like ResNet, BERT, and Transformer.

The development of large models has been strongly influenced by empirical scaling laws. Kaplan and colleagues observed approximate power-law relationships between cross-entropy loss and model size, data, and compute for particular language models, data distributions, and training ranges. This makes resource planning partly predictable within a range, but it does not guarantee proportional improvement in every capability or unlimited scaling of all three resources. Data quality, architecture, optimization, inference cost, and diminishing returns change the effective boundary. GPT-3's 175 billion parameters mark one historical configuration, not proof that larger is necessarily better. See Scaling Laws for Neural Language Models and the GPT-3 paper.

1.1.2 The Core Definition of Modern AI: Data-Driven Learning

In summary, when we talk about AI today, we are mainly talking about data-driven methods represented by machine learning (ML), especially deep learning (DL). The core idea is no longer to pre-program rules and logic by human experts (as in Symbolism), but to design a model with a large number of adjustable parameters, and then let the model automatically learn hidden patterns and regularities from massive data.

This process can be analogized to educating a child. We do not tell the child rigid rules like "an animal with four legs, fur, and that meows is a cat." Instead, we show them thousands of pictures of cats (data), allowing them to form an abstract concept of "cat" in their minds (model). When they see a cat they have never encountered before, they can accurately recognize it -- this is generalization capability.

Therefore, modern AI can be defined as: A field of computer science dedicated to researching and building agents (systems) that can learn from experience (data) and use the acquired knowledge to perform specific tasks or make predictions.

This definition contains three key elements:

• Task: The specific problem the AI system is designed to solve, such as image classification, machine translation, or autonomous driving.
• Experience: Usually exists in the form of large-scale datasets, serving as the source of model learning.
• Performance: Metrics to measure how well the AI system performs the task, such as accuracy, recall, and response time. The goal of learning is to continuously improve task performance by absorbing information from experience.

Understanding this definition of AI reveals that building a modern AI system is essentially an engineering practice centered on data and using computation as a means. Large models are the most advanced and extreme manifestation of this practice at the current technological level. Their demands on infrastructure stem precisely from their almost greedy craving for data and computation.

1.2 A First Glimpse of Machine Learning Algorithms

Machine learning is the core technology and methodology for achieving AI. It studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures, thereby continuously improving their own performance. Based on different learning methods and data types, machine learning algorithms are typically divided into three categories: supervised learning, unsupervised learning, and reinforcement learning.

1.2.1 Supervised Learning

Supervised learning is the most widely used and mature machine learning method. Its core characteristic is that the training data contains explicit "answers," i.e., labels. The model's goal is to learn the mapping relationship from input (features) to output (labels).

Working Principle: It is like a student doing exercises with a teacher. Each problem (input data) has a standard answer (label). The student continuously corrects their problem-solving approach by comparing their answer with the standard one. In supervised learning, the model makes predictions based on input data, then compares the predictions with the true labels to calculate a loss or error. Next, an optimization algorithm (such as gradient descent) adjusts the model's internal parameters to minimize the loss value. This process repeats until the model can make sufficiently accurate predictions on unseen data.

Main Task Types:

• Classification: The prediction target is a discrete category. For example, determining whether an email is spam (binary classification), or identifying whether an image contains a cat, dog, or bird (multi-class classification).
• Regression: The prediction target is a continuous value. For example, predicting a house's price based on its area, location, age, etc., or predicting tomorrow's temperature based on historical weather data.

Typical Algorithms: Linear regression, logistic regression, support vector machines (SVM), decision trees, random forests, and various types of neural networks (e.g., CNN for image classification, RNN/Transformer for sequence data regression or classification).

1.2.2 Unsupervised Learning

In contrast to supervised learning, unsupervised learning training data has no labels. The model must discover structures, patterns, or relationships from the data on its own.

Working Principle: This is like an anthropologist entering an unknown tribe without any guide or dictionary, having to observe people's behavior, language, and interactions to infer the tribe's social structure and customs. Unsupervised learning algorithms attempt to find inherent "structure" in data.

Main Task Types:

• Clustering: Groups samples in the dataset into clusters, where samples within the same cluster are similar to each other, and samples in different clusters are quite different. For example, users can be segmented into "high-value users," "potential users," and "churn-risk users" based on purchasing behavior for targeted marketing.
• Dimensionality Reduction: Reduces high-dimensional data to low-dimensional data while preserving the main information. This can be used for data visualization (e.g., reducing hundreds of dimensions to 2 or 3 for plotting) or as a preprocessing step for other machine learning tasks to remove noise and reduce computation.
• Association Rule Mining: Discovers interesting associations between data items. The classic example is the "beer and diapers" story, where supermarket customers who buy diapers are also likely to buy beer.

Typical Algorithms: K-Means clustering, hierarchical clustering, PCA (Principal Component Analysis), t-SNE, Apriori algorithm.

1.2.3 Reinforcement Learning

Reinforcement learning focuses on how an agent should take a series of actions in an environment to maximize cumulative reward.

Working Principle: The pattern of reinforcement learning is similar to training a pet. When the pet makes a correct action (like "sit"), we give it a reward (like a treat); when it does something wrong, there might be a slight punishment or no reward. Through trial and error, the pet learns how to behave correctly according to commands to get the most rewards. In reinforcement learning, the agent observes a state in the environment, then chooses an action. Based on this action, the environment transitions to the next state and feeds back a reward signal (positive or negative) to the agent. The agent's goal is to learn a policy, a mapping from states to actions, that maximizes the long-term cumulative reward.

Application Scenarios: Reinforcement learning excels in scenarios requiring a series of decisions to achieve a final goal, such as:

• Game AI: AlphaGo defeating the world's top Go players, and AI reaching superhuman levels in complex video games like StarCraft and Dota 2, are masterpieces of reinforcement learning.
• Robot Control: Controlling robotic arms for complex grasping and assembly tasks, or teaching bipedal robots to walk.
• Resource Scheduling and Optimization: Data center energy management, dynamic routing of network traffic, financial trading strategy formulation, etc.

Typical Algorithms: Q-Learning, SARSA, Deep Q-Network (DQN), Policy Gradient, A3C, PPO.

These three learning paradigms are not entirely independent; there are crossovers, such as semi-supervised learning (combining a small amount of labeled data with a large amount of unlabeled data) and self-supervised learning (automatically generating labels from unlabeled data, a core idea in current large model training). However, regardless of the algorithm, they all rely on a large amount of mathematical computation at the bottom, laying the foundation for our subsequent analysis of their hardware requirements.

1.3 Analysis of the Univariate Linear Regression Algorithm

To concretely understand the computational essence of machine learning algorithms, we choose the simplest and most classic supervised learning algorithm -- Univariate Linear Regression -- for detailed analysis. Although simple, its "model definition - loss function - optimization algorithm" three-stage paradigm is a universal pattern running through almost all modern deep learning models.

Suppose we are a real estate agency that has collected sales data for some houses, including size (square meters) and corresponding selling price (ten thousand yuan). We want to build a model that can predict the likely selling price of a new house based on its size.

1.3.1 Model Definition

Our intuition is that the selling price is likely proportional to the size; the larger the size, the higher the price. This relationship can be approximated by a straight line. Mathematically, the equation of a straight line can be expressed as:

$$ \hat{y} = wx + b $$

Here:

• $x$ is the input feature, i.e., the area of the house.
• $\hat{y}$ (y-hat) is the model's predicted output, i.e., the predicted selling price. We use the hat notation to distinguish it from the true price $y$.
• $w$ is the weight, representing the slope of the line. It measures the impact of house size $x$ on the predicted price $\hat{y}$.
• $b$ is the bias or intercept, representing the predicted price when the house size is 0 (unrealistic, but this is the model's baseline).

This equation is our defined model. Our task is to learn and find an optimal pair of parameters $(w, b)$ so that this line best "fits" our data points.

1.3.2 Loss Function

How do we measure whether a line "fits well"? For each actual house data point $(x_i, y_i)$ (where $x_i$ is the size of the $i$-th house and $y_i$ is its true price), our model gives a predicted value $\hat{y_i} = wx_i + b$. The difference between the predicted and true values is the model's error.

$$ \text{Error}_i = \hat{y_i} - y_i $$

We want the overall error of the model across the entire dataset to be as small as possible. Simply summing all errors could be misleading due to positive and negative errors cancelling each other out. Therefore, we typically use Mean Squared Error (MSE) as the loss function. It calculates the average of the squared prediction errors across all samples:

$$ L(w, b) = \frac{1}{N} \sum_{i=1}^{N} (\hat{y_i} - y_i)^2 = \frac{1}{N} \sum_{i=1}^{N} (wx_i + b - y_i)^2 $$

Where $N$ is the total number of samples in the dataset.

The squaring operation has two benefits: first, it ensures the error value is non-negative; second, it amplifies larger errors, making the model more sensitive to "outrageous" predictions.

Summing and averaging synthesize the model's performance across all data points, providing a global, quantifiable evaluation metric.

This loss function $L(w, b)$ is a function of the parameters $w$ and $b$. Our goal thus becomes finding a set of $(w, b)$ that minimizes $L(w, b)$ mathematically. This is known as an optimization problem.

1.3.3 Optimization Algorithm: Gradient Descent

How do we find the $(w, b)$ that minimizes the loss function? One of the most common and important optimization algorithms is gradient descent.

We can imagine the loss function $L(w, b)$ as a two-dimensional valley landscape, where $w$ and $b$ are coordinates on our map, and the value of $L$ is the height at that coordinate. Our goal is to reach the lowest point of the valley.

The strategy of gradient descent is very intuitive:

1. Random Initialization: First, we randomly choose a starting point $(w_0, b_0)$.
2. Calculate Gradient: At the current position, we compute the gradient of the valley landscape at this point. The gradient is a vector pointing in the direction of the steepest increase of the function. In our case, the gradient consists of the partial derivatives of $L$ with respect to $w$ and $b$: $(\frac{\partial L}{\partial w}, \frac{\partial L}{\partial b})$.
3. Update Parameters: We want to go downhill, so we should take a step in the opposite direction of the gradient. The step size is controlled by a hyperparameter called the learning rate, denoted by $\alpha$. A learning rate that is too small makes descent slow; one that is too large might overshoot, causing oscillation or failure to converge. The parameter update rules are as follows:

$$ w_{\text{new}} = w_{\text{old}} - \alpha \frac{\partial L}{\partial w} $$ $$ b_{\text{new}} = b_{\text{old}} - \alpha \frac{\partial L}{\partial b} $$

4. Iterate: We repeat steps 2 and 3, continuously calculating the gradient and updating the parameters. With each iteration, we move a little closer to a lower point in the valley. Theoretically, after enough iterations, we can approach or reach the valley's lowest point, where $(w, b)$ is the optimal solution we seek.

1.3.4 The Essence of the Computation Process

Now, let us string together the entire training process and focus on the computation steps:

Initialization: Randomly generate $w$ and $b$ (two scalars).

Loop (Epochs):

1. Forward Pass: For each sample $x_i$ in the dataset, perform one multiplication and one addition: $\hat{y_i} = w \cdot x_i + b$. If the dataset has $N$ samples, this step requires $N$ multiplications and $N$ additions.

2. Calculate Loss: For each sample, compute the squared error: $(\hat{y_i} - y_i)^2$. Sum all squared errors, then divide by $N$. This step involves $N$ subtractions, $N$ multiplications, $(N-1)$ additions, and 1 division.

3. Backward Pass (Gradient Computation): Using the chain rule from calculus, compute the partial derivatives of the loss function with respect to $w$ and $b$: $$ \frac{\partial L}{\partial w} = \frac{1}{N} \sum_{i=1}^{N} 2(wx_i + b - y_i) \cdot x_i $$ $$ \frac{\partial L}{\partial b} = \frac{1}{N} \sum_{i=1}^{N} 2(wx_i + b - y_i) $$ Computing these two gradient values also involves a large number of multiplications and additions across the entire dataset.

4. Update Parameters: Perform two multiplications ($\alpha \cdot$ gradient) and two subtractions to update $w$ and $b$.

This simple example reveals the core computation pattern of machine learning training: a large number of repetitive, dataset-wide, simple mathematical operations (primarily multiplication and addition).

When we expand from univariate to multivariate linear regression (i.e., using multiple features like size, age, floor, etc., to predict price), the input $x$ and weights $w$ transition from scalars to vectors. Forward propagation becomes a vector dot product operation.

When we enter the field of deep learning, models consist of many layers of neurons, each layer could be a linear transformation (matrix multiplication) followed by a non-linear activation. At this point, the model parameters $w$ become massive matrices, and the input data $x$ is also often organized into matrices or higher-dimensional tensors. Training a deep neural network, the core computation becomes repeatedly performing large-scale matrix multiplication and addition on massive data.

This finding is crucial; it directly determines which type of computing hardware is more suitable for machine learning tasks.

1.4 Special Demands of Machine Learning Algorithms on Computer Hardware

Through the analysis of linear regression, we already know that machine learning, especially deep learning, presents a very unique pattern of computational load. This pattern is distinct from traditional general-purpose computing tasks (such as web browsing, document processing, database queries), thus placing special demands on underlying hardware.

1.4.1 Core Operational Characteristics of Machine Learning Algorithms

We can summarize the operational characteristics of machine learning algorithms as follows:

Massive Parallelism

Whether it is the forward pass computing the predicted value for each sample, or the backward pass computing the gradient contribution for each sample, these computations are independent across samples. This means we can simultaneously perform the same operation on thousands of data samples without waiting for each other's results.

In neural networks, the computation of neurons within the same layer is also independent. A layer with 1024 neurons can be seen as 1024 independent dot product operations, all of which can be executed in parallel.

This characteristic of data parallelism and model parallelism makes algorithms naturally suited for parallel processors with a large number of computing cores.

Compute-Intensive

Training a large model may require trillions or even quadrillions of floating-point operations (FLOPs). For example, per the GPT-3 paper, training its 175-billion-parameter version took roughly 3.14 x 10^23 FLOPs (an estimate based on the empirical formula of parameters times training tokens; different accounting conventions yield slightly different values).

The core operations are matrix multiplication (GEMM: General Matrix-Matrix Multiplication) and convolution. Although mathematically simple, these operations are enormous in scale. A large model's weight matrix may contain billions of elements, and performing a single multiplication with input data requires staggering computation.

High Memory Bandwidth Requirement

No matter how fast the computation, data needs to be fed to it. During training, huge model parameters (weight matrices) and batches of training data (mini-batches) need to be frequently loaded from main memory to compute units.

If the data transfer speed (memory bandwidth) cannot keep up with the speed at which compute units consume data, the compute units will be idle, waiting, causing a severe waste of computing power. This phenomenon is called the "Memory Wall."

Therefore, for AI computing hardware, having extremely high memory bandwidth is as important as having strong computing power.

Tolerance for Reduced Floating-Point Precision

Traditional scientific computing usually requires double-precision floating-point (FP64) to ensure result accuracy. However, research has found that neural networks are robust to noise, and their training and inference processes do not always require such high precision.

Using lower precision, such as single-precision (FP32), half-precision (FP16), or even 8-bit integers (INT8), brings significant benefits:

• Faster computation: Under the same hardware logic, processing lower-precision data is faster. For example, a 32-bit compute unit can theoretically handle two 16-bit operations simultaneously.
• Lower memory footprint: Model parameters and intermediate results occupy half the memory (FP32 -> FP16), allowing training of larger models or using larger batch sizes.
• Lower power consumption: Both data movement and computation energy consumption are reduced.

Therefore, modern AI hardware is typically specially optimized for these mixed-precision computations. For example, NVIDIA's Tensor Core and Google's TPU support formats like bfloat16 (BF16) and FP16.

Having understood these core characteristics, we can evaluate the strengths and weaknesses of different types of processors in handling machine learning tasks.

1.4.2 Implementing Machine Learning on CPU and Parallel Acceleration

The CPU is the core of the computer we are most familiar with. It is designed as a general-purpose, low-latency processor.

Architecture Characteristics:

• Few but powerful cores: Typical server CPUs have dozens (e.g., 32, 64, 128) of very powerful and complex computing cores.
• Complex control logic: Each core excels at handling complex instruction streams, including extensive branch prediction, out-of-order execution, etc., to optimize single-threaded performance.
• Large caches: CPUs have multiple levels of high-speed cache (L1, L2, L3) to store frequently used data, compensating for the access speed of main memory (DRAM).
• Optimized for serial tasks: The CPU's design philosophy is to complete one complex task after another as quickly as possible.

Performance in Machine Learning:

For small-scale models and data, or algorithms with complex logic and many branches (such as decision trees), the CPU is entirely adequate.

However, when facing deep learning, a massive parallel computing task, the CPU's architectural bottleneck becomes apparent. Its dozens of cores are a drop in the bucket compared to the millions or billions of parallel operations required by neural networks. Most of the time, the vast majority of computing tasks are queued, waiting to be processed by a few cores.

CPU Parallel Acceleration Attempts:

SIMD (Single Instruction, Multiple Data): This is the CPU's primary means of parallel computing. Modern CPUs support SIMD instruction sets like SSE and AVX. AVX-512 instructions can perform the same operation on a 512-bit vector (e.g., 16 FP32 or 32 FP16 floats) in a single clock cycle. Using specially optimized math libraries (like Intel MKL-DNN, OpenBLAS), a certain degree of parallel acceleration can be achieved on the CPU.

Multi-threading/Multi-core Parallelism: Utilize multiple CPU cores to split tasks (such as processing a mini-batch) across different cores. This needs to be implemented through multi-threading programming frameworks like OpenMP or TBB.

Conclusion: Although software optimization can squeeze out the CPU's parallel potential, its "general-purpose" design intent means it is far less efficient at handling "highly specialized" large-scale parallel computing workloads than hardware designed specifically for that purpose. The CPU is better suited as a "general manager," responsible for operating system operation, data preprocessing, task scheduling, and other control-intensive tasks, offloading the compute-intensive "heavy lifting" to more specialized coprocessors.

1.4.3 The Main Engine for Machine Learning Algorithms -- GPU

The rise of the GPU is the most important hardware foundation for the deep learning revolution. Interestingly, GPUs were not originally designed for AI.

Architecture Characteristics:

• Many-Core Architecture: Contrary to CPUs, GPUs have thousands (e.g., NVIDIA A100 has 6,912 CUDA cores) of relatively simple, low-power computing cores.
• Optimized for Parallel Tasks: The GPU's design philosophy is real-time 3D graphics rendering. Rendering a single frame requires performing similar shading, transformation, and other calculations on millions of pixels on the screen. The calculations for these pixels are independent of each other, inherently a large-scale parallel task. The GPU's architecture is born to efficiently handle such tasks.
• High Throughput Design: The GPU's design goal is to maximize throughput, i.e., the total work completed per unit time, rather than pursuing low latency like the CPU (the speed of completing a single task).
• Extremely High Memory Bandwidth: To feed thousands of computing cores, GPUs are typically equipped with dedicated, very high-bandwidth memory (such as GDDR6X, HBM2e, HBM3), whose bandwidth far exceeds the DDR memory connected to the CPU.

From Graphics Rendering to General-Purpose Computing (GPGPU):

Researchers keenly noticed that the GPU's parallel computation pattern for processing pixels is strikingly similar to many algorithms in scientific computing (like physics simulations, signal processing), especially linear algebra operations. A matrix element can be seen as a pixel on the screen.

NVIDIA's introduction of CUDA in 2007 was a revolutionary milestone. It provides a complete programming model, API, and toolchain, allowing developers to write general-purpose parallel computing programs directly on the GPU using languages similar to C/C++, without the cumbersome approach of simulating graphics APIs (like OpenGL, DirectX). This greatly lowered the barrier to GPGPU and opened the era of GPUs as general-purpose computing accelerators.

The Perfect Match between GPU and Deep Learning:

Deep learning's core operation -- large-scale matrix multiplication -- perfectly matches the GPU's architectural strengths. A large matrix multiplication can be decomposed into thousands of small, independent dot product operations, then distributed to thousands of GPU cores for simultaneous execution.

The GPU's high memory bandwidth ensures that huge model parameters and data can be quickly fed to compute cores, avoiding idle computing power.

To further accelerate AI workloads, modern GPUs also integrate dedicated hardware units. For example, NVIDIA introduced Tensor Core starting with the Volta architecture. This is a hardware circuit specifically designed for mixed-precision matrix multiply-accumulate operations (D = A * B + C), capable of completing a 4x4 FP16 matrix multiplication with FP32 accumulation in a single clock cycle, with theoretical peak performance far exceeding ordinary CUDA cores. This gives GPUs exponentially improved efficiency when executing deep learning training and inference tasks.

Conclusion: With its large-scale parallel computing architecture, high memory bandwidth, and dedicated hardware for AI computations (like Tensor Core), the GPU has become the undisputed main engine for current machine learning, especially large model training and inference tasks. Building a large model computing center largely means building a computing cluster composed of hundreds or thousands of high-end GPU cards working efficiently together.

1.4.4 The New Engines for Machine Learning Algorithms -- TPU and NPU

When a computation task's pattern becomes sufficiently fixed and important, designing an ASIC dedicated to it becomes a natural choice. ASICs are "custom-tailored" for a specific task, able to discard all irrelevant general-purpose logic, achieving peak performance and energy efficiency. TPUs and NPUs are products of this line of thinking.

TPU:

The TPU is an ASIC specifically designed by Google to accelerate its internal neural network workloads (from search and translation to advertising).

Its core innovation is the Systolic Array. This is a 2D grid of a large number of simple processing units (multiply-accumulate units, MACs). Data flows through the array like blood in a heart, "pulsing" at a fixed rhythm. Weight parameters are pre-loaded into the MAC units of the array, while input data is fed into the array one clock cycle at a time. At each clock cycle, each MAC unit performs one multiply-accumulate operation and passes the result to neighboring units.

The advantages of this architecture are:

• Extremely high compute density: A very large systolic array can be integrated on a single chip (e.g., TPUv3 has two 128x128 systolic arrays).
• Extremely high energy efficiency: Data flows regularly within the chip, greatly reducing the need for high-power memory reads and writes. Most of the time, data is passed directly between compute units.

TPU design is entirely dedicated to neural network computation. It is extremely efficient at performing large-scale matrix multiplication but has limited capability in handling general-purpose computation or complex control flow. Google has built powerful, specialized AI supercomputers for training and inference by constructing Pods composed of thousands of TPUs.

NPU:

• NPU is a broader concept, generally referring to any processor used to accelerate neural network operations. TPU is one type of NPU, but NPU also includes various designs from other vendors.
• NPUs are often integrated into edge devices, such as smartphones (e.g., Apple's Neural Engine, Qualcomm's Hexagon DSP), smart cameras, and domain controllers for autonomous vehicles.
• Unlike data-center-grade TPUs that pursue extreme peak performance, edge-side NPUs focus more on energy efficiency (Performance per Watt). They need to efficiently execute inference tasks within limited power and thermal budgets.

The designs of various NPUs are diverse, but their core idea is similar: by hard-wiring common neural network operators like convolution, matrix multiplication, and activation functions in hardware, they achieve performance and efficiency orders of magnitude higher than CPUs or general-purpose DSPs.

Conclusion: TPUs and NPUs represent the "specialization" direction of AI hardware development. By customizing hardware for the core computation patterns of neural networks, they achieve unparalleled performance and energy efficiency. Although less general than GPUs, they are extremely powerful new engines in their areas of expertise. In the future, we are likely to see a new computing paradigm where CPUs (for control), GPUs (for general parallel computing and some AI tasks), and NPUs/TPUs (for heavy AI tasks) work together in a heterogeneous computing system.

1.5 Chapter Summary

As the foundation of the entire book, this chapter has taken the reader on a cognitive journey from macro to micro, aiming to establish a fundamental understanding of the infrastructure requirements of the large model era.

We first reviewed the tortuous yet glorious development history of AI, clarifying that the AI we talk about today is fundamentally based on data-driven machine learning, with deep learning and large models being the most advanced achievements of this paradigm.

Next, through a preliminary introduction to the three major machine learning paradigms -- supervised learning, unsupervised learning, and reinforcement learning -- we understood the problem-solving approaches of different algorithms. To delve into the computational essence, we analyzed in detail the most basic case of univariate linear regression. This simple example clearly revealed the common computation pattern underlying all complex machine learning algorithms: intensive computation dominated by multiply-accumulate operations, on a massive, highly parallel scale, over vast amounts of data.

Finally, we used this core computation characteristic as a "yardstick" to measure and analyze different types of computing hardware. We recognized:

The CPU, as a general-purpose processor, with its "few but strong" cores optimized for serial execution, struggles to efficiently handle the "human-wave tactics" of AI workloads. It is better suited for the role of the system's "brain."

The GPU, with its "many-core" parallel architecture and high-bandwidth memory originally designed for graphics rendering, formed a "perfect match" with deep learning computation patterns. Through the maturity of GPGPU and the CUDA ecosystem, and the enrichment of dedicated hardware like Tensor Core, it has become the absolute mainstay for current AI computation, especially large model training.

TPUs, NPUs, and other ASICs represent the ultimate "specialization" form of hardware development, hard-wiring the core operations of neural networks into hardware circuits to pursue extreme performance and energy efficiency, constituting indispensable new engines in the AI computing ecosystem.

Through the study of this chapter, we should deeply realize that the infrastructure requirements of large models are not simply about piling up servers. They are systematic, structural demands -- determined by the inherent characteristics of the algorithms -- for large-scale parallel computing capability, high memory and network bandwidth, and heterogeneous computing collaboration. This lays a solid theoretical foundation for our subsequent chapters' in-depth exploration of specific technical details such as GPU hardware, server design, network and storage architecture, and cloud-native platforms. Starting from the next chapter, we will officially enter these exciting technical worlds.