祝融说。

In the first two parts, we successfully built a solid, cloud-native AI infrastructure. We have a compute pool of top-tier GPUs and NPUs, laid down RDMA high-speed networks, built high-performance storage, and deployed Volcano/Yunikorn platforms capable of intelligently scheduling AI tasks. We have a modern factory with "sophisticated hardware and intelligent software."

Now, it is time to truly make this production line run, to produce its ultimate product -- the large language model.

In this chapter, we will formally shift our role, partly from "platform builder" to "production operator." We will delve into the full process of large model training, like an experienced head chef who not only understands the ingredients (data) but also masters various cooking techniques (training modes) and can skillfully direct a large kitchen team (distributed parallelism). We will first dissect from a macro perspective the three most important training modes in the LLM lifecycle: Pre-training, Full Fine-tuning, and Efficient Fine-tuning (using LoRA as an example), understanding their different resource requirements. Next, from a unique operations perspective, we will deeply analyze the three parallel strategies that make trillion-parameter model training possible: Data Parallelism, Tensor Parallelism, and Pipeline Parallelism. Finally, we will combine theory and practice, concluding the chapter with a highly challenging and realistic case study -- deploying a full fine-tuning task for a mainstream open-source large model on a Huawei Ascend cluster -- providing you with an actionable and reproducible operational guide.

6.1 Training Mode Analysis: Pre-training vs. SFT vs. LoRA

"Training a large model" is a very broad statement. In the actual LLMOps workflow, "training" can be divided into at least three distinctly different modes based on purpose and resource consumption. As an AI Infra engineer, understanding their differences is crucial because it directly determines your resource planning, scheduling strategy, and cost accounting.

6.1.1 Pre-training: Casting the Model's "Soul"

Pre-training is the process of training a foundation model from scratch on an ultra-large-scale, high-quality, general-purpose text dataset (typically TB or even PB level).

Goal: To teach the model the general rules of human language, grammatical structures, factual knowledge, and some reasoning ability. It is not about completing a specific task, but about building an "embryo" with broad "general knowledge."

Analogy: Pre-training is like the general education phase of a person's life, from birth to university graduation. They read tens of thousands of books in the library (massive data), learning language, history, science, and various subjects, but have not yet entered any specific professional field.

Infrastructure Requirements ("Three Highs and One Long"):

1. Massive Compute: This is the most compute-intensive stage. Training a hundred-billion-parameter model typically requires thousands of top-tier training cards (like H800 or 910B).
2. High Bandwidth Network: A cluster of this scale must use InfiniBand or high-performance RoCE networks, combined with NCCL/HCCL for efficient collective communication; otherwise, communication overhead will drown out computation gains, resulting in poor linear scalability.
3. High Throughput Storage: A high-performance parallel file system (like Lustre/GPFS) is needed to continuously, with high throughput, "feed" training data to thousands of compute nodes.
4. Long Duration: The pre-training process is extremely long, typically lasting weeks or even months. This imposes extreme requirements on the stability and fault tolerance of the entire cluster. Extended downtime can mean losses on the order of millions of dollars (depending on cluster scale and compute prices). Therefore, robust checkpoint mechanisms, hardware health inspections, and fault self-healing capabilities are essential.

Operations Perspective Focus:

• MFU and Linear Scalability: These are the "golden metrics" for the pre-training phase. The operations team's core KPI is to maximize these two metrics by any means possible. Even a 5% improvement means saving days of time and enormous costs.
• Mean Time to Recovery (MTTR): Due to the long cycle, hardware failures (e.g., GPU falling off the bus, optical module damage) are inevitable. Quickly detecting problems, locating them, replacing hardware, and resuming training from the latest checkpoint is the most important daily task for the operations team.
• Cost Control: During the planning phase, the compute estimation models we will detail in the next chapter must be used to precisely calculate the required resources and time, and to develop a budget.

6.1.2 SFT (Supervised Fine-Tuning): Teaching the Model to "Speak Like a Human"

The pre-trained foundation model, while knowledgeable, only knows how to "continue writing" text, not how to "converse." It does not know how to follow instructions or act as a useful AI assistant. SFT is the process that solves this.

Goal: Using a much smaller but extremely high-quality dataset of "instruction-response pairs" to teach the model how to understand and follow human instructions.

Analogy: SFT is like the onboarding training a university graduate (foundation model) receives upon joining a company. By learning a large number of "question-standard answer" cases (instruction dataset), they learn how to communicate and work as a qualified "employee."

Infrastructure Requirements ("Medium Scale, High-Frequency Iteration"):

1. Medium Compute: SFT datasets typically contain only tens of thousands to hundreds of thousands of records, far smaller than pre-training. The required compute scale is correspondingly lower, typically in the range of tens to hundreds of cards.
2. Reduced Network and Storage Requirements: Due to the smaller cluster size, the extreme performance requirements for networking and storage are somewhat relaxed, but good RDMA networking and stable storage are still needed.
3. High-Frequency Iteration: Algorithm engineers will frequently experiment with different datasets, hyperparameters, and model versions. SFT task cycles are typically hours to days.
4. Full-parameter Fine-tuning: In the SFT phase, by default, all parameters of the model (e.g., 70 billion parameters) undergo gradient updates. This means its per-GPU memory requirement is exactly the same as in the pre-training phase!

Operations Perspective Focus:

• Job Scheduling and Resource Turnover: The core challenge for the operations team is how to efficiently schedule these high-frequency, medium-scale SFT tasks, ensuring that the algorithm team's experiments can start and complete quickly, maximizing GPU turnover. Queue management, priority settings, and fair scheduling become especially important.
• Memory Footprint: This is the most problematic aspect of SFT (full fine-tuning). A 70B model, even when fine-tuned with a small amount of data, occupies the same amount of memory for model weights, gradients, and optimizer states as when pre-training with massive data. Therefore, precise calculation of memory requirements and selection of appropriate parallel strategies are crucial. We will detail the calculation methods in the next chapter.
• Experiment Management: Needs to integrate with MLOps platforms (like MLflow, WandB) to help algorithm engineers track the configuration, data, code, and results of each experiment.

6.1.3 PEFT (Parameter-Efficient Fine-Tuning), Using LoRA as an Example: A Model "Minimally Invasive Surgery"

While full fine-tuning (SFT) is effective, its cost remains high. Updating tens of billions of parameters each time not only requires many high-end GPUs but also results in massive storage waste from saving a complete model copy for each downstream task. PEFT, especially LoRA, has revolutionized this problem.

LoRA (Low-Rank Adaptation) Core Idea:

• During training, freeze the pre-trained model's original weights completely.
• Alongside certain layers of the model (typically the Attention layers of a Transformer), inject two small, trainable "bypass" matrices (called A and B). The "rank" of these matrices is very low, meaning their parameter count is extremely tiny.
• During fine-tuning, only the parameters of these two small matrices are trained.
• During inference, the trained matrices A and B are multiplied, and the result is added to the original weight matrix, thereby adapting the model's behavior without changing the original model.

Goal: To achieve results comparable to full fine-tuning at a fraction of the cost, adapting the model to specific downstream tasks or domain styles.

Analogy: LoRA is like giving an experienced general practitioner (foundation model) a "minimally invasive procedure" or a "specialized addendum." The doctor's own ability does not change, but through this small adjustment, they can effectively handle cases in a specific domain (e.g., pediatrics or cardiology).

Infrastructure Requirements ("Low Barrier, Mass Personalization"):

1. Extremely Low Compute: The number of parameters that LoRA needs to train is typically a very small fraction of the original model (depending on the rank setting; it can be below one percent in order of magnitude). This means a 70B model that previously required 8 A100s to fine-tune can now potentially be done on a single consumer-grade card (like an RTX 4090), depending on rank, sequence length, and batch settings.
2. Sharply Reduced Memory Requirements: Since there is no need to store the massive optimizer state (only for LoRA parameters), memory requirements are drastically reduced.
3. Fast Training: LoRA training typically takes only tens of minutes to hours.
4. Flexible Deployment: The output LoRA weight file is only tens of MB, not hundreds of GB. For deployment, one base model plus multiple LoRA "plugins" can be loaded on demand, enabling large-scale personalized services.

Operations Perspective Focus:

• Inference Service Optimization: LoRA makes it possible to dynamically load and switch model adapters on the inference side. The operations team needs to focus on how the inference server (e.g., vLLM, SGLang) integrates and optimizes LoRA, such as efficient LoRA weight merging and batching.
• Fragmented Resource Management: LoRA tasks have small, scattered resource requirements, which may generate a lot of "scraps" of GPU resources. How to use MIG (NVIDIA) or similar technologies to partition a physical GPU for multiple LoRA training or inference tasks becomes a new challenge for improving utilization.
• Version Management: Managing thousands of tiny LoRA weight files and their correspondence with base models imposes new requirements on MLOps version control and asset management.

Summary and Comparison:

As an AI Infra engineer, you need to quickly assess the resource profile based on the type of task submitted by the algorithm team and match it with the most suitable cluster, queue, and scheduling strategy.

6.2 Distributed Parallelism Strategies: Data Parallelism, Tensor Parallelism, and Pipeline Parallelism from an Operations Perspective

When a model cannot fit in one GPU and the target cannot be met by shrinking the model, quantization, gradient checkpointing, parameter-efficient training, or CPU/NVMe offload, distributed training is usually necessary. If a single GPU is merely too slow, compare the speedup against communication and operational cost first. Common methods include data, tensor, and pipeline parallelism, which can be combined into multidimensional parallelism.

From an operations and resource management perspective, understanding the principles, communication patterns, and resource requirements of these three parallel strategies is fundamental to performance optimization and troubleshooting.

6.2.1 Data Parallelism (DP): The Simplest, Most Common "Human Wave Tactic"

Core Idea:

1. Model Replication: Copy the complete model to every GPU participating in training.
2. Data Splitting: Split a large training batch into N parts (N is the number of GPUs), with each GPU receiving one mini-batch.
3. Independent Computation: Each GPU independently performs forward and backward propagation, computing its own gradients.
4. Gradient Synchronization: This is the key and bottleneck of DP. All GPUs aggregate their gradients (typically by averaging) through one All-Reduce collective communication operation.
5. Synchronous Update: All GPUs use the aggregated global gradients to update their own model copies in exactly the same way, ensuring all models are consistent at the start of the next iteration.

Analogy:

DP is like having a class of students (GPUs) work on the same set of exercises (model). The teacher divides the exercises into N parts (data splitting), and each student does a part. After finishing, everyone brings their answers (gradients) to the blackboard for comparison, calculates the standard answer (All-Reduce), and then each person corrects their own paper based on the standard answer (model update).

Key Points from an Operations Perspective:

• Communication Overhead: DP's communication overhead is proportional to the model's parameter count. The larger the model, the more gradient data needs to be transmitted per All-Reduce. Therefore, DP is very sensitive to network bandwidth.
• Memory Bottleneck: DP cannot solve the problem of insufficient GPU memory. Because each GPU needs to hold the complete model, gradients, and optimizer states. If a model itself cannot fit into a single card, using DP alone is ineffective.
• Best Suited For: Scenarios where the model can fit on a single card, but you want to accelerate training (shorten time) by adding more GPUs. This is the most common and basic parallel approach. PyTorch's DistributedDataParallel (DDP) is its standard implementation.
• Optimization: ZeRO (Zero Redundancy Optimizer)
  • ZeRO technology developed by DeepSpeed is a major optimization of DP. It shards and distributes the three most memory-intensive components -- model parameters, gradients, and optimizer states -- across all GPUs, drastically reducing the peak memory per card.
  • For example, in ZeRO-3, each GPU stores only a part of the model parameters. During computation, it dynamically retrieves the required parts from other GPUs via All-Gather. ZeRO makes it possible to train very large models on cards with limited memory, but at the cost of increased communication volume.

6.2.2 Tensor Parallelism (TP): "Dismembering" the Model

Core Idea:

• When a model's weight matrix (tensor) itself is too large to fit into a single GPU's memory, TP comes into play. It splits a single large matrix operation within the model across multiple GPUs for collaborative computation.
• Take a standard Transformer layer as an example: its core consists of Self-Attention and Feed-Forward Network (FFN), both involving extensive matrix multiplication.
• TP splits these large matrices (e.g., the projection matrices for Q, K, V) by rows or columns, distributing them across different GPUs.
• Each GPU holds and computes only a portion of the matrix. During computation, they need to exchange intermediate results through collective communication operations like All-Gather or Reduce-Scatter to assemble the final complete output.

Analogy:

TP is like having multiple students (GPUs) collaboratively compute a huge matrix multiplication. One student is responsible for computing the left half of the result matrix, another for the right half. During the computation, they need to exchange some intermediate row or column data with each other.

Key Points from an Operations Perspective:

• Solving Single-GPU Memory Bottleneck: This is TP's core value. If a model has P parameters and uses N-way TP, each GPU only needs to store approximately P/N model parameters, reducing the memory requirement to 1/N.
• Communication Pattern: TP communication occurs within each forward and backward pass, very frequently. It is extremely sensitive to network latency. Therefore, TP is typically confined within a single node, utilizing the fastest NVLink/HCCS for communication. TP across nodes is rare, as cross-node network latency would severely drag down computation.
• Complex Implementation: TP requires deep modification of the model code. Not all models can easily undergo tensor parallelism. Megatron-LM is a representative framework for implementing TP.

6.2.3 Pipeline Parallelism (PP): The Model "Factory Assembly Line"

Core Idea:

• Distribute different layers of the model across different GPUs.
• For example, a 48-layer model using 4-way PP. GPU 0 is responsible for layers 1-12, GPU 1 for layers 13-24, and so on.
• Data (mini-batches) flows sequentially through each GPU, like on a factory assembly line. After GPU 0 finishes computing the first 12 layers, it sends its output (called activations) to GPU 1. GPU 1 continues computation based on this and passes it to GPU 2.
• Pipeline Bubble: Naive PP is inefficient. At the beginning and near the end of the pipeline, most GPUs are idle. For example, when GPU 0 is processing the first mini-batch, all other GPUs are idle. This idle time is the "pipeline bubble."

Optimization: GPipe / PipeDream

• To reduce the bubble, modern PP implementations (like GPipe) split a large batch into multiple smaller micro-batches.
• GPU 0 processes the first micro-batch, immediately passes it to GPU 1, and then immediately starts processing the second micro-batch.
• In this way, after a brief "warm-up" period, all GPUs can process different micro-batches simultaneously, the pipeline is "filled," and parallel efficiency is greatly improved.

Analogy:

PP is like a car assembly line. GPU 0 installs the chassis, GPU 1 installs the engine, GPU 2 installs the body, and GPU 3 paints the car. By splitting a large order (Batch) into multiple cars (micro-batches) and sending them continuously down the production line, every workstation (GPU) stays busy.

Key Points from an Operations Perspective:

• Solving Memory Bottleneck: Similar to TP, PP also splits the model, reducing per-GPU memory. If a model has L layers and uses N-way PP, each GPU only needs to store approximately L/N layers, roughly reducing the memory requirement to 1/N.
• Communication Pattern: PP communication is point-to-point, occurring only between adjacent GPUs on the pipeline. The amount of data communicated is the size of the activations for each micro-batch.
• Load Balancing is Key: PP efficiency is highly dependent on load balancing. If the computation assigned to each GPU (number of layers, layer complexity) is uneven, faster GPUs will wait for slower ones, creating new bubbles. How to reasonably split the model is a problem requiring profiling and tuning.
• Memory Wall: PP incurs additional memory overhead because each GPU needs to cache activations for multiple micro-batches for use during backward propagation.

6.2.4 3D Parallelism: The Ultimate Combination

When training ultra-large models in the trillion-parameter range (e.g., GPT-4, PaLM), a single parallel strategy is often insufficient. The industry uses a "3D Parallelism" strategy that combines all three:

• Data Parallelism (DP): At the outermost layer, the entire training task is replicated multiple times to process more data and accelerate convergence.
• Pipeline Parallelism (PP): In the middle layer, the massive model is split vertically and distributed across different compute nodes.
• Tensor Parallelism (TP): At the innermost layer, within each node, the computation of a single layer is split horizontally across all GPUs on that node.

A Typical 3D Parallelism Configuration Example:

• Total Resources: 1024 GPUs, distributed across 128 nodes (8 cards per node).
• TP Size: 8-way (using NVLink within each node for tensor parallelism).
• PP Size: 16-way (splitting the model into 16 segments, distributed across 16 different node groups).
• DP Size: 8-way (there are a total of 8 such 128-card (16x8) complete model replicas performing data parallelism).
• Total Size: 8 * 16 * 8 = 1024.

As an AI Infra engineer, when faced with such a complex 3D parallel task, you need to be able to clearly analyze its communication patterns and perform targeted optimization on the infrastructure:

• TP communication relies on intra-node NVLink bandwidth.
• PP communication relies on inter-node RDMA network bandwidth and latency.
• DP communication consumes large amounts of cross-node RDMA network bandwidth.

A bottleneck in any single link will affect the final performance of the entire "3D barrel."

6.3 Case Study: Deploying a Full Fine-Tuning Task for Llama 3 / Qwen on an Ascend Cluster

Theory must ultimately be grounded in practice. This section will tackle a scenario of great practical significance in the current context of domestic technology adoption: performing full supervised fine-tuning (SFT) on a mainstream open-source large model (e.g., Llama 3-8B or Qwen-14B) on a K8s cluster based on Huawei Ascend 910B, using Volcano for scheduling.

This case study will weave together many of the knowledge points from this part and even the entire book: Ascend hardware, CANN software stack, K8s Device Plugin, Volcano scheduler, distributed parallelism strategies, etc.

6.3.1 Preparation: Environment and Software Stack

Hardware Environment (synthetic example):

• Assume we have a K8s cluster containing several compute nodes, each equipped with 8 Ascend 910B NPUs.
• Nodes are interconnected via a 100G RoCE v2 network.
• The cluster has a high-performance parallel file system (e.g., Lustre) deployed and mounted on all nodes at /mnt/data.

Software Environment (K8s Side):

• The K8s cluster has the Ascend Device Plugin installed and configured, allowing K8s to recognize huawei.com/npu resources.
• The K8s cluster has the Volcano scheduler installed and configured.
• The cluster's image registry has the base image ready.

Building the Base Image (Dockerfile)

We need a base image containing PyTorch for Ascend, HCCL, the Transformers library, and DeepSpeed.

# Use Huawei's official PyTorch for Ascend base image
FROM ascendhub.huawei.com/public-ascendhub/pytorch-ascend:2.1.0-cann7.0.1-py39-ubuntu20.04-x86_64

# Install necessary tools and dependencies
RUN apt-get update && apt-get install -y git vim wget

# Upgrade pip and install core Python packages
RUN pip install --upgrade pip
# DeepSpeed needs adaptation for Ascend; typically compiled from a specific source branch or using Huawei's provided version
# This is an example; please refer to Huawei's official documentation
RUN pip install deepspeed==0.1x.x-ascend
RUN pip install transformers==4.3x.x
RUN pip install accelerate

# Clean up cache
RUN rm -rf /root/.cache/pip

WORKDIR /workspace

Note: The adaptation of libraries like DeepSpeed and Transformers for Ascend is constantly evolving. Please refer to Huawei's official documentation or its CodeLab/Gitee community for the correct installation method compatible with your CANN version.

Data and Model Preparation:

• Model Weights: Download the Llama 3-8B or Qwen-14B model weights from Hugging Face or a domestic mirror source, and store them on the shared filesystem, e.g., /mnt/data/models/Llama-3-8B.
• Fine-Tuning Data: Prepare an SFT dataset, typically in JSON lines format, with each line containing an instruction and the expected output, e.g., the Alpaca dataset. Store it at /mnt/data/datasets/alpaca_data_zh.json.

6.3.2 Adapting and Writing the Training Script

We will use the DeepSpeed framework to simplify distributed training configuration. DeepSpeed integrates well with the Transformers Trainer API.

DeepSpeed Configuration File (ds_config.json)

This is one of the most critical configuration files. It tells DeepSpeed which parallelism strategy and optimization to use. For full fine-tuning, we will use ZeRO-2 to optimize memory.

{
    "train_micro_batch_size_per_gpu": 4,
    "gradient_accumulation_steps": 2,
    "optimizer": {
    "type": "AdamW",
    "params": {
        "lr": 2e-5,
        "betas": [0.9, 0.999],
        "eps": 1e-8
    }
    },
    "fp16": {
    "enabled": true,
    "loss_scale_window": 1000
    },
    "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "reduce_bucket_size": 5e8,
    "contiguous_gradients": true
    }
}

Training Launch Script (run_sft.sh)

This script is responsible for calling the deepspeed command with all necessary parameters.

#!/bin/bash

# Get distributed information from Volcano environment variables
# Volcano injects these automatically, and HCCL reads them
export MASTER_ADDR=${VC_WORKER_0_HOST}
export MASTER_PORT=${VC_WORKER_0_PORT}
export RANK=${VC_TASK_INDEX}
export WORLD_SIZE=${VC_WORKER_NUM}

# DeepSpeed launch command
deepspeed --num_gpus=8 train_script.py \
    --deepspeed ds_config.json \
    --model_name_or_path /mnt/data/models/Llama-3-8B \
    --data_path /mnt/data/datasets/alpaca_data_zh.json \
    --output_dir /mnt/data/outputs/llama3-8b-sft-v1 \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 2 \
    --save_strategy "steps" \
    --save_steps 500 \
    --learning_rate 2e-5 \
    --fp16 True

Key Point: train_script.py is a standard training script written using the Transformers Trainer API. The magic of DeepSpeed is that you hardly need to modify this Python script; you only need to inject distributed capability through the command line and ds_config.json.

6.3.3 Submitting to K8s: The VolcanoJob YAML

Finally, we package everything into a VolcanoJob and submit it to the K8s cluster. Assume we need 2 nodes, totaling 16 NPUs.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: llama3-8b-sft-job
spec:
  schedulerName: volcano
  minAvailable: 2 # 2 Pods, 8 cards each, total 16 cards
  queue: high-priority-gpu
  tasks:
    - name: worker
      replicas: 2
      template:
        spec:
          containers:
            - name: trainer
              image: my-registry/my-ascend-app:1.0
              command: ["/bin/bash", "-c", "./run_sft.sh"]
              resources:
                limits:
                  huawei.com/npu: 8 # Each Pod requests 8 NPUs
              volumeMounts:
                - name: data-storage
                  mountPath: /mnt/data
          restartPolicy: OnFailure
          volumes:
            - name: data-storage
              persistentVolumeClaim:
                claimName: my-pvc-for-lustre # Mount shared storage

Workflow Breakdown:

1. User runs kubectl apply -f job.yaml.
2. Volcano scheduler receives this Job with 2 Pods, minAvailable is 2.
3. The scheduler checks resources in the high-priority-gpu queue and finds 2 nodes each with 8 free NPUs.
4. Pre-check passes; Volcano starts scheduling. It places pod-0 on node-A and pod-1 on node-B.
5. Kubelet creates pod-0 on node-A. The Ascend Device Plugin is called to inject the environment variables and devices for 8 NPUs into the container.
6. The container in pod-0 starts, executing run_sft.sh. The script reads environment variables injected by Volcano (like MASTER_ADDR, which is pod-0's IP).
7. pod-1 starts on node-B in the same way.
8. The HCCL library in both Pods, based on the environment variables, successfully establishes communication. A 16-card distributed training task is now running! The checkpoints produced during training are saved in /mnt/data/outputs.

6.3.4 Operations and Troubleshooting

• Pod Pending: If the Job remains Pending for a long time, use kubectl describe podgroup <job-name> to view Volcano's events, which will tell you why scheduling failed (e.g., insufficient resources).
• Training Stuck: If the training log has no output for a long time, it is likely an HCCL communication issue. Check the RoCE network configuration between nodes, and verify that pod-0 can ping pod-1.
• OOM (Out of Memory): If the log reports insufficient memory, try reducing train_micro_batch_size_per_gpu in ds_config.json, or upgrading from ZeRO-2 to ZeRO-3 (but this will increase communication volume).

This case study completely demonstrates how to operate a complex large model training task end-to-end on a real, domestically-produced AI infrastructure. It is not only a comprehensive test of your technical skills but also the ultimate reflection of your value as a top-tier AI Infra engineer.