祝融说。

In Part I, we completed the foundation of the intelligent computing center. We delved into the core of AI chips, laid down RDMA high-speed networks, and built parallel storage systems. Now, we have powerful but raw bare-metal compute power. This is like owning a well-equipped factory with no standardized production lines. Deploying AI applications directly on bare-metal servers would quickly plunge us into "dependency hell" -- Python version conflicts between projects, incompatible system libraries, and environments that are difficult to migrate and reproduce. This is unacceptable in the fast-iterating AI era.

The answer is containerization. Container technology (represented by Docker) packages an application and all its dependencies into a lightweight, portable "shipping container," achieving good isolation and environment consistency. It is the core technology we need to build standardized production lines for the AI factory.

However, getting AI applications to "live in" containers is not trivial. There is a natural gap between standard containers and the underlying GPU/NPU hardware. This chapter will guide you step by step in bridging this gap. We will first solve the most fundamental problem: how to build a container environment that can "see" and use GPUs and NPUs. Next, we will integrate this capability into the grand narrative of Kubernetes, allowing K8s to intelligently schedule and manage AI compute just as it schedules CPUs. Finally, we will impart the unique skills of "image engineering," teaching you how to build lightweight, efficient, and cross-platform standardized images for large models. Mastering this chapter will enable you to build the first and most critical pillar of a cloud-native AI platform.

4.1 Building the Foundation: NVIDIA Container Toolkit and Ascend Docker Runtime

4.1.1 The Root of the Problem: The Container's "Blindness"

Let us start with a simple experiment. On a host with NVIDIA drivers correctly installed, open a terminal, type nvidia-smi, and you will see detailed GPU information. Now, let us start a standard Ubuntu container and try to execute the same command inside it:

# Execute on the host, successfully displays GPU information
$ nvidia-smi

# Start a standard Ubuntu container
$ docker run --rm -it ubuntu:20.04 /bin/bash

# Execute inside the container, command not found or error
root@container:/# nvidia-smi
bash: nvidia-smi: command not found

Why does this happen? The reason lies in the core isolation mechanism of containers: Linux namespaces. Containers have their own independent filesystem, process space, and network stack. They are isolated from the host. By default, they cannot "see" the host's device files (e.g., /dev/nvidia0) or driver libraries (e.g., libcuda.so).

To break this isolation and allow containers to access GPUs and NPUs, we need a "middleman" or "interpreter." This role is played by NVIDIA's Container Toolkit and Huawei's Ascend Docker Runtime.

4.1.2 NVIDIA Container Toolkit: Injecting CUDA Power into Containers

The NVIDIA Container Toolkit is a suite of components that extends the standard container runtime (e.g., Docker's runc) to be aware of and utilize NVIDIA GPUs.

Core Component Analysis

1. libnvidia-container: A core library that provides the low-level API for interacting with the NVIDIA driver, responsible for querying GPU information, configuring the container environment, etc.
2. nvidia-container-cli: A command-line tool called by the container runtime. It uses libnvidia-container to prepare the GPU environment.
3. nvidia-container-runtime: The critical "interpreter." It is a custom OCI (Open Container Initiative) runtime that intercepts standard container creation requests. When it detects that a request requires a GPU, it calls nvidia-container-cli to dynamically mount the necessary NVIDIA driver files, device nodes, and library files into the container's namespace, and then calls the standard runtime (runc) to start the container.

Workflow

1. The user executes docker run --gpus all ....
2. The Docker Daemon receives the request, sees the --gpus parameter, and knows that a GPU is needed.
3. Instead of directly calling the default runc, the Docker Daemon calls nvidia-container-runtime.
4. nvidia-container-runtime takes over the request. It calls nvidia-container-cli to query the host for available GPU devices and the location of driver libraries.
5. nvidia-container-cli automatically adds these device files (e.g., /dev/nvidia0, /dev/nvidia-uvm) and library files to the container's configuration.
6. Finally, nvidia-container-runtime calls runc with this "enhanced" configuration to create and start the container.
7. As a result, when the container starts, it already has everything it needs to access the GPU.

Installation and Configuration

1. Ensure the NVIDIA driver is installed. This is a prerequisite for everything.
2. Add the NVIDIA software repository

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

3. Install the Toolkit

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

4. Configure and restart Docker Daemon

The installation script usually configures Docker automatically. You can verify the settings in /etc/docker/daemon.json and ensure nvidia-container-runtime is set as the default runtime. Then restart the Docker service:

sudo systemctl restart docker

5. Verify the installation

docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi

If the GPU information is printed successfully, your basic environment is ready!

The Power of Environment Variables:

The Toolkit also injects several key environment variables into the container for fine-grained GPU control:

• NVIDIA_VISIBLE_DEVICES: This is the most important one. Its value determines which GPUs are "visible" inside the container. It can be all, or a list of GPU indices (e.g., 0,1) or UUIDs. docker run --gpus '"device=0,1"' ultimately results in NVIDIA_VISIBLE_DEVICES=0,1 inside the container.
• NVIDIA_DRIVER_CAPABILITIES: Controls which driver libraries are mounted into the container. The default is compute,utility; graphics applications may need graphics.

4.1.3 Ascend Docker Runtime: The Containerization Cornerstone for the Ascend Platform

Following a similar approach to NVIDIA, Huawei also provides a mechanism for containers to use Ascend NPUs. This mechanism is typically provided as part of the CANN (Compute Architecture for Neural Networks) software package.

Core Components and Principles

• Ascend Docker Runtime: Also a custom Docker runtime. Its role is to mount the host's CANN drivers, firmware, necessary libraries, and NPU device files (e.g., /dev/davinci0, /dev/devmm_svm) into the container when it starts.
• Difference from NVIDIA: Compared to NVIDIA Toolkit's independent installation, the installation and configuration of Ascend Docker Runtime are more tightly coupled with the CANN Toolkit installation process.

Installation and Configuration

1. Ensure the CANN software package is correctly installed. This includes the driver, firmware, and toolkit.
2. Install Ascend Docker Runtime: The CANN package usually includes an ascend-docker-runtime deb or rpm package.

# Assuming you are in the CANN installation directory
sudo dpkg -i ascend-docker-runtime_*.deb

3. Configure Docker Daemon

Edit /etc/docker/daemon.json to add the Ascend runtime, and possibly set it as the default.

{
    "runtimes": {
    "ascend": {
        "path": "/usr/local/bin/ascend-docker-runtime",
        "runtimeArgs": []
    }
    },
    "default-runtime": "ascend"
}

4. Restart Docker service

sudo systemctl restart docker

5. Verify the installation

When starting a container, you need to use the --device parameter to manually specify the NPU devices to map.

docker run -it --device=/dev/davinci0 --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc ascendhub.huawei.com/public-ascendhub/mindspore-ascend:2.2.11-cann7.0.1-py39-euleros2.10-aarch64 npu-smi info

If NPU 0 information is displayed, the configuration is successful.

Environment Variables: Ascend containers also rely on environment variables to specify which devices to use. The most critical is ASCEND_VISIBLE_DEVICES, which functions identically to NVIDIA_VISIBLE_DEVICES.

Summary: Whether for NVIDIA or Huawei Ascend, the core idea of their containerization solutions is the same: provide a privileged, hardware-aware custom runtime that "interferes" with the standard container startup process to "smuggle" the host's drivers and devices into the isolated container environment. Completing this step paves the way for managing AI compute in K8s.

4.2 The K8s Device Plugin: How to Make K8s "See" and Allocate GPUs and NPUs

Running containers with --gpus or --device on a single machine is just the first step. In a real intelligent computing center, we face a massive cluster of hundreds or thousands of servers. We cannot manually ssh into each machine to execute docker run. We need a "central brain" -- Kubernetes -- to uniformly schedule and manage these valuable AI compute resources.

The problem: K8s natively only knows about CPU and Memory resources. How does it know that node-01 has 8 H800 GPUs while node-02 has 8 910B NPUs? The answer lies in the K8s Device Plugin framework.

4.2.1 Device Plugin: K8s's "Hardware Interpreter"

Device Plugin is a standard, open extension mechanism provided by K8s. It allows third-party hardware vendors to register their proprietary hardware (e.g., GPUs, NPUs, FPGAs, high-performance NICs) as first-class resources in the K8s cluster, making them schedulable and requestable.

Work Mode:

• A Device Plugin is typically a Pod running as a DaemonSet on every compute node (or nodes with specific labels).
• It communicates with the Kubelet on that node via a defined gRPC interface. The communication uses a Unix socket file typically located at /var/lib/kubelet/device-plugins/kubelet.sock.

4.2.2 The Device Plugin Lifecycle (Core Principles)

Understanding the Device Plugin workflow is key to understanding how K8s manages AI compute. The entire process can be broken down into three steps:

Step 1: Discovery and Registration

• When a Device Plugin Pod starts on a node, it first scans the node, discovering the AI accelerator cards and their IDs by calling native tools like nvidia-smi or npu-smi.
• It then connects to the Kubelet's gRPC service and calls the Register method.
• In this call, it tells the Kubelet: "Hello, I am a plugin from NVIDIA. I provide a new resource called nvidia.com/gpu. Please note this." (Huawei's plugin would register a resource like huawei.com/npu).
• Upon receiving the registration, the Kubelet is aware of this new resource.

Step 2: Reporting and Monitoring (ListAndWatch)

• After successful registration, the Kubelet calls back the Device Plugin's ListAndWatch method.
• The Device Plugin immediately returns a list of all currently available device IDs on that node, e.g., [GPU-UUID-1, GPU-UUID-2, ...].
• Upon receiving this list, the Kubelet updates the Node object's status.capacity and status.allocatable fields, writing information like nvidia.com/gpu: 8.
• It is at this moment that the K8s "central brain" (API Server and Scheduler) truly "sees" those 8 GPUs!
• ListAndWatch is a streaming RPC. The Device Plugin continuously monitors the hardware state. If a GPU goes offline due to a fault, it immediately informs the Kubelet through this stream. The Kubelet updates the node's allocatable resources accordingly, preventing the scheduler from scheduling tasks onto a malfunctioning card.

Step 3: Request and Allocation (Allocate)

Now, a user can submit a Pod YAML file, requesting GPU resources in resources.limits:

resources:
    limits:
    nvidia.com/gpu: 2

• During scheduling, Kube-scheduler iterates through all nodes, searching for nodes whose status.allocatable for nvidia.com/gpu is greater than or equal to 2. It then schedules the Pod to one of those nodes.
• After the Pod arrives at the target node, before creating the container, the Kubelet calls the Device Plugin's Allocate method again, telling it: "This Pod needs 2 GPUs. Please tell me the allocation result."
• The Device Plugin selects 2 cards from its maintained list of free devices (e.g., IDs GPU-UUID-3 and GPU-UUID-5) and returns a ContainerAllocateResponse. This response contains the necessary information to start the container:
  • Devices: The device file paths that need to be mounted into the container, e.g., /dev/nvidia3, /dev/nvidia5.
  • Envs: The environment variables to be injected into the container. The most important one is NVIDIA_VISIBLE_DEVICES=3,5.
• The Kubelet passes this information to the underlying container runtime (Docker + NVIDIA Container Toolkit). The Toolkit, based on NVIDIA_VISIBLE_DEVICES=3,5, precisely exposes only the 3rd and 5th GPUs to the container.
• At this point, a Pod requesting specific GPUs is successfully running.

4.2.3 Practical Deployment

Deploying the NVIDIA Device Plugin:

NVIDIA provides Helm Charts and YAML files, making deployment very simple.

# Install using Helm
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm install \
    --generate-name \
    --set-string nodeSelector.accelerator=nvidia-h800 \
    nvdp/nvidia-device-plugin

Note the nodeSelector here; it ensures the NVIDIA plugin only runs on nodes with NVIDIA GPUs.

Deploying the Ascend Device Plugin:

Huawei also provides YAML deployment files for its plugin, typically included in the CANN software package or its open-source community.

# Example DaemonSet snippet
apiVersion: apps/v1
kind: DaemonSet
metadata:
    name: ascend-device-plugin-daemonset
spec:
    template:
    spec:
        nodeSelector:
        accelerator: ascend-910b # Ensure it only runs on Ascend nodes
        containers:
        - name: ascend-device-plugin
        image: ascend-device-plugin:latest
        # ... mount necessary sockets and device directories

Summary: Device Plugin is a model example of the "open and extensible" design philosophy of Kubernetes. Through an elegant gRPC protocol, it abstracts a wide variety of hardware into unified, declarable resources. It is the cornerstone of unified heterogeneous compute scheduling. As an AI Infra engineer, you must not only know how to deploy it but also deeply understand the core flow of Register -> ListAndWatch -> Allocate. This will give you a clear line of reasoning when troubleshooting issues like "Why won't the Pod schedule?" or "Why was the GPU allocated incorrectly?"

4.3 Image Engineering: Slimming and Multi-Stage Building for Docker Images in Heterogeneous AI Environments

We can now schedule Pods that require GPUs or NPUs on K8s. Now we face another equally tricky problem: where do the container images for these Pods come from?

For LLM applications, the dependency environment is extremely complex: specific versions of CUDA or CANN, specific versions of PyTorch, massive Python dependency packages, plus the model weights themselves. An unoptimized "naive" image can easily exceed 20 GB. In an LLMOps workflow requiring frequent updates and rapid deployment, such a "behemoth" image is catastrophic:

• Slow Pull Times: When a new node starts up in the cluster, pulling a 20 GB image could take tens of minutes, severely impacting auto-scaling efficiency.
• Expensive Storage: Storing hundreds of versions, each 20 GB, in the image registry is a significant cost.
• Poor Security: The image contains many unnecessary build tools and libraries, increasing the attack surface.

Image Engineering is the "art" of transforming bloated AI application images into lightweight, efficient, secure, and maintainable ones.

4.3.1 Core Principle 1: Choose the Right "Foundation" -- Base Image

Choosing a good base image is the first and most important step in image slimming.

• Avoid using -devel images for production:

  • NVIDIA's nvidia/cuda image tags typically come in three varieties: -base, -runtime, and -devel.
  • -devel contains the full CUDA compiler (nvcc), header files, and debugging tools. It has the largest size and should only be used for the compilation stage.
  • -runtime contains only the runtime libraries necessary for running CUDA programs. It is much smaller.
  • -base is even more stripped down, potentially not even including common libraries like cuDNN.
  • Principle: The final production image must be built from a -runtime or -base image.

• The Ascend ecosystem equivalent: Huawei's base images (e.g., mindspore-ascend) also have different versions. Choose a version that contains only the CANN runtime environment, not the complete development toolkit, as your production base.

4.3.2 Core Principle 2: "Two-Phase Construction" -- Multi-Stage Builds

This is the "nuclear weapon" of image slimming. The idea is to use multiple FROM instructions in a single Dockerfile, dividing the build process into multiple stages. Only the final stage's output becomes the final image.

Scenario: We have a custom PyTorch C++ extension that needs compilation.

Bad Dockerfile (Single Stage):

# Image is huge (e.g., 15 GB)
FROM nvidia/cuda:12.1.0-devel-ubuntu22.04

# Install build tools and Python
RUN apt-get update && apt-get install -y g++ python3-pip

# Install Python dependencies
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

# Compile and install custom extension
COPY . .
RUN python3 setup.py install

CMD ["python3", "main.py"]

This image contains the g++ compiler, the complete CUDA SDK, pip cache, and all other "construction waste."

Good Dockerfile (Multi-Stage):

# --- Stage 1: Builder ---
# Use a development image for compilation
FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder

RUN apt-get update && apt-get install -y g++ python3-pip python3.10-venv
WORKDIR /app

# Install in a virtual environment for easy packaging
RUN python3 -m venv /app/venv
ENV PATH="/app/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .
RUN python3 setup.py install

# --- Stage 2: Final ---
# Use a lightweight runtime image
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04

# Copy only the necessary runtime files and compiled artifacts
COPY --from=builder /app/venv /app/venv
COPY --from=builder /app/main.py /app/main.py

ENV PATH="/app/venv/bin:$PATH"
WORKDIR /app

CMD ["python3", "main.py"]

The final generated image might be only 5 GB in size. It contains no compile-time dependencies like g++ or the CUDA SDK -- only the clean Python virtual environment and the final runtime script.

4.3.3 Core Principle 3: "Be Frugal" -- Optimize Dockerfile Instructions

Combine RUN Instructions: Each RUN, COPY, or ADD instruction in a Dockerfile creates a new image layer. Too many layers increase image size and build time.

# Bad: creates multiple layers
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# Good: combine into one layer and clean up promptly
RUN apt-get update && \
    apt-get install -y curl && \
    rm -rf /var/lib/apt/lists/*

Leverage Build Cache: Docker caches the result of each layer. Place infrequently changing instructions (like installing system packages) early, and frequently changing ones (like COPY source code) later, to maximize cache usage and speed up subsequent builds.

Use .dockerignore: Create a .dockerignore file in the project root to exclude files that do not need to be copied into the image (e.g., .git directory, test data, local configuration files). This reduces the build context size and prevents sensitive information leaks.

4.3.4 Challenge: Image Strategy in Heterogeneous Environments

When your cluster contains both NVIDIA and Ascend hardware, how do you manage application images?

Strategy 1 (Recommended): Separate Images

Build two separate images for the same application, with clear tags:

• myapp:1.2.0-cuda12.1
• myapp:1.2.0-cann7.0

In K8s deployment files (e.g., Deployment or Job), use nodeSelector to decide which image to use:

# Pod template for NVIDIA nodes
spec:
    nodeSelector:
    accelerator: nvidia-h800
    containers:
    - name: main
    image: my-registry/myapp:1.2.0-cuda12.1

# Pod template for Ascend nodes
spec:
    nodeSelector:
    accelerator: ascend-910b
    containers:
    - name: main
    image: my-registry/myapp:1.2.0-cann7.0

• Advantages: Images are clean, minimal, and have no redundancy. This is the cleanest approach and most aligned with cloud-native principles.
• Disadvantages: Requires maintaining two CI/CD pipelines.

Strategy 2 (Advanced, Use with Caution): Fat Image with Runtime Detection

Build a single "fat image" containing both CUDA and CANN dependencies.

Modify the container's entry point script (ENTRYPOINT) to detect the current environment at startup and choose the correct execution path.

#!/bin/bash
if [ -d "/usr/local/cuda" ]; then
    echo "CUDA environment detected. Starting application for NVIDIA."
    # Execute NVIDIA version start command
    exec python3 main_cuda.py "$@"
elif [ -d "/usr/local/ascend" ]; then
    echo "Ascend/CANN environment detected. Starting application for Huawei."
    # Execute Ascend version start command
    exec python3 main_ascend.py "$@"
else
    echo "Error: No supported AI accelerator environment found."
    exit 1
fi

• Advantages: Only one image tag to manage.
• Disadvantages: The image is extremely bloated, violates the single responsibility principle, and is more complex to manage and debug. Generally not recommended unless there are very special unified delivery requirements.

Summary: In this chapter, we started from scratch and successfully packaged AI applications into standardized containers that can be uniformly scheduled in K8s. We mastered the "runtime" technology for connecting containers to hardware, understood the "device plugin" principles for making K8s hardware-aware, and learned the "image engineering" art of building lightweight, efficient images. At this point, we have a solid "foundation" for our cloud-native AI platform. Next, we will build a smarter, more efficient scheduling and resource management system on top of this foundation.