祝融说。

In the previous chapter, we pushed the performance of a single GPU to its absolute limit. Through techniques like quantization and vLLM, we learned how to make an LLM run faster and more efficiently on a single machine. This might be sufficient for individual developers or small-scale projects. However, in a large enterprise, a cloud service provider, or any scenario serving millions of users, the challenges we face grow exponentially.

Imagine these scenarios:

• A team of hundreds of algorithm engineers needs to share a cluster of thousands of GPUs (an illustrative scale for teaching). How do you allocate and isolate resources fairly and efficiently?
• The flagship AI product of a company may see daytime peak inference traffic one to two orders of magnitude higher than the overnight trough (illustrative figures for teaching). How do you achieve elastic scaling of services to handle the flood peaks while saving costs during off-peak hours?
• We need to simultaneously provide dozens of different versions and sizes of LLM inference services for multiple business lines. How do you manage, monitor, and iterate these models uniformly without falling into "operations hell"?

These issues go far beyond the scope of single-machine optimization. They require us to elevate our perspective from "individual combat" to "army-level coordination," from focusing on the performance of a single process to focusing on the elasticity, reliability, and cost-effectiveness of the entire AI infrastructure.

In this chapter, we will enter the highest level of AI engineering — large-scale deployment and cluster management. This is no longer solely the responsibility of an algorithm engineer, but a challenge jointly faced by AI platform engineers, SREs (Site Reliability Engineers), and DevOps engineers. However, understanding the logic and tools behind this, as a striving AI engineer, will open the door for you to become a system architect and technology leader.

Together, we will explore:

• Overview of AI Infrastructure: We will build a macro-level understanding from a single machine to a cluster, understanding why clusters are needed and the new challenges they bring.
• Kubernetes and KubeFlow: We will learn the de facto standard for container orchestration — Kubernetes (K8s) — and KubeFlow, which is specifically designed for machine learning workflows. You will understand how they achieve resource scheduling, service discovery, and fault self-healing.
• GPU Resource Scheduling and Utilization Optimization: The GPU is the most expensive and scarce resource in AI infrastructure. We will explore how GPU sharing and virtualization technologies break the limitation of "one card for one task," substantially improving GPU utilization (the exact magnitude depends on the workload profile; this is a directional statement, not a promised figure).
• Architecture Design of Model as a Service (MaaS): We will take a higher-level perspective and think about how to design an enterprise-grade "Model as a Service" platform. This platform needs to manage the entire lifecycle of models (from training and evaluation to deployment) and provide standardized, highly available model inference APIs for upper-layer businesses.

This chapter's content is profound and broad. It will reveal the full picture of the "iceberg" that supports giant AI applications like ChatGPT and Midjourney. Mastering this knowledge will enable you to design and build truly industrial-grade AI systems capable of supporting massive-scale businesses. Now, let us move from the familiar single-machine environment into the magnificent world of cloud-native AI clusters.

11.1 Overview of AI Infrastructure: From Single Machine to Cluster

11.1.1 Limitations of Single-Machine Deployment

Everything we have done in the previous chapters can basically be accomplished on a single powerful server with 1-8 GPUs. Single-machine deployment is simple and direct, making it ideal for development, experimentation, and small applications. However, as the business scale grows, its limitations quickly become apparent:

1. Resource Bottlenecks and Waste:

   Limits of Scale-up: A single server can only accommodate a limited number of GPUs. When you need to train a super-large model requiring 32 or even hundreds of GPUs, a single machine is insufficient. Resource Idleness: In an AI team, Zhang San trains a model during the day, occupying 8 GPUs. Li Si runs inference tests at night, needing only 1 GPU. During each person's working hours, the other's resource demands cannot be met, and for most of the day, some GPUs on the server remain idle, causing huge resource waste. Resource Conflicts Between Tasks: If Zhang San and Li Si run tasks on the same machine simultaneously, they may interfere with each other due to VRAM contention or CUDA version conflicts, potentially causing task failures.

2. Lack of Elasticity:

   Production inference traffic typically fluctuates. A service deployed on a single machine has fixed capacity. To handle traffic peaks, you must configure hardware for peak demand, but during low traffic periods, this hardware sits idle. We cannot dynamically increase or decrease service instances based on real-time load.

3. Lack of High Availability:

   Single-machine deployment has a single point of failure. If the server's hardware (such as power supply or motherboard) fails, or if the operating system crashes, the entire AI service goes down until manual repair is completed. This is unacceptable for online services that need to run 7x24.

11.1.2 Moving to a Cluster: The Inevitable Choice for Distributed Computing

To overcome the above limitations, we must connect multiple physical servers together to form a cluster. A cluster is a collection of multiple computers managed by a unified software system, appearing to the user as a single, massive "supercomputer."

Benefits of a Cluster:

1. Resource Pooling: The cluster pools the CPU, memory, GPU, and other resources of all servers into a single huge "resource pool." Administrators and scheduling systems can allocate resources on demand for different users and tasks, achieving unified resource management and efficient utilization.
2. Scale-out: When resources are insufficient, instead of buying more expensive single servers, we can simply add more standardized server nodes to the cluster. This scaling method is cheaper and theoretically has no limit.
3. High Availability and Fault Recovery: The cluster management system continuously monitors the health status of all nodes and services. When a node or service instance fails, it can automatically restart a new instance on another healthy node, achieving fault self-healing and ensuring business continuity.
4. Elastic Scaling: The cluster can automatically scale out or scale in the number of service replicas based on real-time load metrics (such as CPU utilization, request queue length), achieving true on-demand resource usage.

New Challenges Introduced by Clusters:

Of course, clusters also introduce new complexity:

• Resource Scheduling: How do you decide which node a task (such as a training job or inference service) should run on? How do you satisfy its requirements for specific GPU models and VRAM sizes?
• Service Discovery and Load Balancing: A service may have multiple replicas running on different nodes with different IP addresses. How do clients find these services? How is traffic evenly distributed among these replicas?
• State Management and Storage: How do you manage the state of distributed tasks? How do you provide reliable distributed storage for applications that need to persist data?
• Network Communication: How do you ensure efficient, low-latency network communication between nodes within the cluster?

Solving these complex distributed system problems is the core mission of container orchestration systems like Kubernetes.

11.2 Kubernetes and KubeFlow in AI Scenarios

11.2.1 Kubernetes (K8s): The Operating System of the Cloud-Native Era

Kubernetes (often shortened to K8s) is an open-source system for automating the deployment, scaling, and management of containerized applications. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation (CNCF). You can think of K8s as the "operating system of the data center."

In the K8s world, a container (usually a Docker container) is the basic unit of application deployment. As we learned in Chapter 4, containers package an application and all its dependencies together, achieving environment consistency. K8s manages the lifecycle of these containers across the entire cluster.

K8s Core Concepts (from an AI scenario perspective):

• Pod: The smallest deployable unit in K8s. A Pod can contain one or more closely related containers. For example, a Pod for an inference service might contain a main container running vLLM and a "sidecar" container for log collection.
• Deployment: Defines the "desired state" of a service, e.g., "I want 3 replicas of the LLM inference service running." K8s' controller continuously works to keep the actual state of the cluster consistent with this desired state. If a Pod goes down, Deployment automatically creates a new one to replace it.
• Service: Provides a single, stable access point (a virtual IP address) and load balancing for a group of functionally identical Pods. No matter how the backend Pods are created, destroyed, or moved, clients only need to access this Service's address.
• Node: A physical or virtual machine in the cluster. K8s' Scheduler is responsible for assigning Pods to appropriate Nodes to run on.
• YAML: K8s uses a declarative API. Users describe the desired resource state by writing YAML files, then submit them to the K8s cluster using the kubectl apply -f my-app.yaml command.

How K8s Empowers AI Workloads:

By containerizing our training jobs or inference services and managing them with K8s, we automatically gain all the benefits of clusters mentioned earlier: resource pooling, elastic scaling, and high availability. For example, we can configure a Horizontal Pod Autoscaler (HPA) so that K8s automatically increases or decreases the number of inference service Pods based on GPU utilization.

11.2.2 KubeFlow: An All-in-One K8s Package Tailored for Machine Learning

While K8s provides general-purpose container orchestration capabilities, it does not inherently understand the specialized domain of "machine learning." For example, it does not know what a "training job," "hyperparameter search," or "model version" is.

KubeFlow is an open-source platform built on top of K8s, specifically designed for machine learning workflows. Its goal is to make deploying, scaling, and managing complex ML systems on K8s simple, portable, and scalable.

KubeFlow is not a single piece of software, but a "suite" of independent components, each addressing a specific problem in the ML lifecycle:

• KubeFlow Pipelines: Used for building and managing end-to-end ML workflows. You can define a complex ML process (e.g., data preprocessing -> model training -> model evaluation -> model deployment) as a Directed Acyclic Graph (DAG), where each node is a containerized step. KubeFlow Pipelines handles executing these steps in order on K8s and managing dependencies and data transfer between them.
• TF-Operator / PyTorch-Operator: Provide native support for distributed training. You only need to define your training roles (such as Master, Worker, Parameter Server) in a YAML file. These Operators automatically create the corresponding Pods in K8s and configure the network communication between them, allowing you to easily run large-scale distributed training on K8s.
• Katib: A component for hyperparameter tuning and Neural Architecture Search (NAS).
• KServe (formerly KFServing): A component specifically designed for model inference services. It provides out-of-the-box features such as automatic scaling (including scaling to zero), model version management, canary deployments, and request/response logging, greatly simplifying the complexity of model deployment.

The Relationship Between K8s and KubeFlow: K8s is the foundation, providing underlying resource management and container orchestration capabilities. KubeFlow is the "ML platform suite" built on top of that foundation, leveraging K8s capabilities and encapsulating a large number of higher-level abstractions and tools oriented toward ML scenarios, allowing algorithm engineers to focus more on models and algorithms themselves without needing to understand the underlying details of K8s.

11.3 GPU Resource Scheduling and Utilization Optimization

The GPU is the most valuable and expensive resource in an AI cluster. Yet, under traditional usage patterns, GPU utilization is often shockingly low. A typical scenario is a developer applying for a V100 GPU for code debugging or running a small experiment. This task might only use 10% of the GPU's compute units and 20% of its VRAM, but the card is exclusively occupied by that task for its entire duration, preventing anyone else from using it.

To solve this problem, a range of GPU virtualization and sharing technologies have emerged.

11.3.1 NVIDIA MPS (Multi-Process Service)

MPS is an NVIDIA technology that allows multiple CUDA processes to run simultaneously and in parallel on the same GPU.

How It Works: MPS starts a daemon process that manages CUDA contexts from different processes. When multiple processes submit compute tasks to the GPU, the MPS server aggregates the kernels from these tasks and executes them in parallel on the GPU's Streaming Multiprocessors (SMs).

Advantages: Can significantly improve GPU utilization when handling a large number of small, concurrent compute tasks.

Disadvantages: All processes share the GPU's VRAM and compute resources without VRAM isolation. An OOM (Out of Memory) error in one process can affect all other processes. It also cannot limit the maximum VRAM or compute power a process can use.

11.3.2 MIG (Multi-Instance GPU)

MIG is a hardware-level virtualization technology introduced by NVIDIA on high-end GPUs based on the Ampere architecture (such as A100) and later.

How It Works: MIG allows a single physical GPU to be partitioned at the hardware level into up to 7 independent GPU instances (GI). Each GI has its own dedicated compute engines, VRAM, and memory bandwidth, and they are fully isolated from each other.

Advantages:

• Strong Isolation: A failure or load spike in one GI does not affect any other GI. This provides isolation guarantees similar to having a physical GPU.
• QoS (Quality of Service) Guarantees: Can allocate one or more GIs to each task, ensuring it has exclusive access to fixed compute and VRAM resources.

Disadvantages:

• Hardware Limitation: Only supported on high-end data center GPUs like A100, H100, etc.
• Fixed Partition Granularity: The partitioning methods and resource sizes for each GI are predefined and not very flexible.

11.3.3 GPU Time-Slicing

This is a software-level sharing mechanism implemented by the K8s scheduler or third-party plugins (such as NVIDIA's GPU Operator).

How It Works: Allows multiple Pods to declare that they need a GPU, even if the total number of GPUs requested exceeds the number of physical GPUs on the node. The scheduler performs time-slicing scheduling among these Pods. At any one point in time, only one Pod's process can actually access the GPU; other processes are waiting.

Advantages: Simple to implement, allows over-subscription of GPU resources.

Disadvantages: Cannot run in parallel, not suitable for latency-sensitive inference tasks. More suitable for offline tasks or development environments that can tolerate interruptions and latency.

11.3.4 GPU Virtualization Technologies (e.g., cGPU, gpushare)

This is currently the most flexible and mainstream GPU sharing solution in cloud-native AI platforms. Taking Alibaba Cloud's cGPU technology as an example:

How It Works: It achieves fine-grained splitting and isolation of GPU compute power and VRAM by modifying the NVIDIA driver and container runtime. When requesting a Pod, users can request 0.3 of a GPU, or 5Gi of VRAM, just like requesting CPU resources.

Advantages:

• Fine-Grained Resource Control: You can allocate any proportion of compute power and any amount of VRAM on demand, achieving "request what you get."
• Isolation: Provides isolation at both the VRAM and compute power levels. A Pod cannot use more than its requested resources, preventing resource contention.
• Maximized Utilization: A single physical GPU can be shared simultaneously by multiple tasks with different resource requirements (e.g., a training task needing high compute but low VRAM, and an inference task needing low compute but high VRAM), thereby maximizing overall GPU utilization.

Selection Summary:

• Development/Testing Environment: Time-slicing or GPU virtualization is an excellent choice for improving resource utilization.
• High-Performance Inference/Training: For tasks requiring exclusive, stable performance, allocate full GPUs or use MIG.
• Mixed Loads / Multi-Tenant Platforms: GPU virtualization technologies (such as cGPU) offer the best flexibility and resource utilization, making them ideal for building enterprise-grade AI platforms.

11.4 Architecture Design of Model as a Service (MaaS)

When an enterprise needs to manage and provide dozens or hundreds of AI models, building a separate deployment, monitoring, and operations system for each model would be a disaster. Model as a Service (MaaS) aims to solve this problem by building a unified, platform-oriented approach.

The goal of a MaaS platform is to allow algorithm engineers to "self-service" deploy their models as highly available online services, without worrying about the complex details of K8s, GPU scheduling, network configuration, and so on.

A typical MaaS platform architecture usually consists of the following core layers:

1. Infrastructure Layer

   • Compute Resources: A cluster of GPU servers, either physical or in the cloud.
   • Container Orchestration: With Kubernetes at its core, responsible for underlying resource scheduling and container management.
   • GPU Management: Integrates GPU drivers, NVIDIA Operator, and GPU virtualization/sharing solutions to achieve pooling and fine-grained scheduling of GPU resources.
   • Storage: Provides distributed file systems (such as Ceph, NFS) for storing model files and datasets, and object storage (such as S3) for unstructured data.

2. Model Management and Serving Layer

   • Model Registry: A centralized system for storing and version-managing all models. Similar to Docker Hub for container images. Each model has a unique name and version number, along with its metadata (such as source, performance metrics, input/output formats, etc.).
   • Inference Serving Engine: The core of the MaaS platform. It is typically based on KServe or a custom control plane, integrating high-performance inference frameworks (such as vLLM, TensorRT-LLM). It is responsible for:
     • Pulling the specified version of the model from the Model Registry.
     • Creating a Deployment for the inference service on K8s based on user configuration (such as replica count, GPU resources, quantization strategy).
     • Configuring the service's auto-scaling policy (HPA).
     • Providing lifecycle management for the service (deploy, retire, version update).
   • Unified API Gateway: All requests to model inference services enter through a unified API Gateway. The Gateway handles:
     • Authentication and Authorization: Verifying the caller's identity and permissions.
     • Routing: Forwarding requests to the correct backend model service based on the request's URL or headers.
     • Rate Limiting and Circuit Breaking: Protecting backend services from being overwhelmed by traffic spikes.
     • Request/Response Logging.

3. Operations and Monitoring Layer

   • Observability:
     • Monitoring: Using tools like Prometheus to collect and store metrics for systems and applications, such as GPU utilization, VRAM usage, QPS, latency, and number of model service replicas.
     • Logging: Using solutions like ELK (Elasticsearch, Logstash, Kibana) or Loki to centrally collect and query all service logs.
     • Tracing: Using Jaeger or OpenTelemetry to trace the complete call chain of a request across distributed systems, facilitating the location of performance bottlenecks.
   • Alerting: Based on metrics collected by Prometheus, configure alerting rules (e.g., "when P99 latency exceeds 500ms") and send notifications via Alertmanager.
   • CI/CD (Continuous Integration/Continuous Deployment): Automating the model training, evaluation, packaging, and deployment process. For example, when a new model version meets certain criteria during evaluation, the CI/CD pipeline can automatically register it in the Model Registry and trigger a canary deployment, gradually shifting a small amount of traffic to the new version, observing its performance, and then rolling out to full production.

MaaS Platform Workflow:

1. Algorithm Engineer: Trains a model and pushes it (or its LoRA weights) along with a configuration file to a Git repository.
2. CI/CD Pipeline: Triggered by the Git commit. Automatically runs tests, builds the model container image, and pushes the model file to the Model Registry.
3. Deployment: The algorithm engineer submits a deployment request to the MaaS platform via a Web UI or YAML file, specifying the model name, version, required resources, etc.
4. MaaS Platform: Receives the request, pulls the model from the Model Registry, creates the inference service Deployment on the K8s cluster, and configures network routing and monitoring.
5. Business Units: Call the newly deployed model service through the unified API Gateway.
6. Operations/Platform Engineers: Monitor the resource usage of the entire platform and the health status of all model services through a unified monitoring dashboard (such as Grafana).

By building such a MaaS platform, enterprises can greatly improve the iteration speed and deployment efficiency of AI applications, reduce operations costs, and achieve fine-grained management and maximum utilization of precious GPU resources.

Chapter Summary

In this chapter, we elevated our perspective from single-machine performance optimization to the strategic level of building and managing large-scale, production-grade AI infrastructure.

We first understood the necessity of moving from a single machine to a cluster, clarifying the significant advantages of clusters in resource pooling, elasticity, and high availability, as well as the distributed system challenges that come with them.

Then, we learned about Kubernetes, the de facto standard of the cloud-native era, and KubeFlow, the "all-in-one suite" designed for machine learning. We understood how they simplify and automate complex AI workflows through container orchestration and high-level abstractions.

We delved into the most critical cost and bottleneck in AI infrastructure — GPU resource management. We learned about a range of technologies, from MPS, MIG, to time-slicing and GPU virtualization, mastering how to break the exclusivity of GPU usage and push resource utilization to the extreme.

Finally, from the perspective of a system architect, we designed an enterprise-grade Model as a Service (MaaS) platform. We understood how such a platform integrates complex functions like model management, inference services, monitoring, and operations through a layered design, thereby achieving automated management of the AI model lifecycle and providing stable, efficient, and scalable model capabilities for upper-layer businesses.

Completing this chapter means you have developed a full-stack AI engineering perspective from the micro to the macro level. You can not only write efficient algorithm code and optimize single-point inference performance, but also design and understand the complex backend systems that support the entire AI business. This comprehensive ability spanning applications, platforms, and infrastructure is the key to excellence and becoming a technology leader in the field of AI engineering. At this point, we have completed the study of all core technical chapters of this book. In the final chapter, we will review the growth path of an AI engineer and look ahead to future technology trends.