祝融说。

In the previous chapter, we successfully integrated AI compute power (GPU/NPU) into Kubernetes via the Device Plugin mechanism and learned how to build standardized container images. At this point, our K8s cluster has the basic capability to run AI tasks. However, a new and more severe challenge is quietly approaching, originating in the "central brain" of the K8s cluster -- the Scheduler.

Imagine a busy airport. The control tower (K8s Scheduler) is responsible for assigning planes (Pods) to appropriate gates (Nodes). For ordinary web applications, "passenger planes," scheduling is relatively simple: as long as a gate is available, the plane can dock. But AI training tasks, especially distributed training, are more like a "space shuttle launch mission" that requires multiple gates to coordinate simultaneously. If the control tower still schedules the space shuttle the same way it schedules passenger planes, chaos and disaster are inevitable.

In this chapter, we will deeply analyze the "misfit" of the native K8s scheduler in AI scenarios and introduce the "advanced control towers" designed specifically for these "space shuttle missions" -- Volcano and Yunikorn. We will deploy and configure them hands-on and delve into the core advanced scheduling strategies behind them: Gang Scheduling, Binpack, and Topology-Aware Scheduling. Mastering these concepts will allow you to truly manage resource allocation in large-scale AI clusters and become a qualified "AIDC tower commander."

5.1 Why the Native K8s Scheduler is Unsuitable for AI (Deadlock and Fragmentation Issues)

Kubernetes's default scheduler (kube-scheduler) is an excellent general-purpose scheduler. Its core logic is: process Pods one by one, finding the best node for each Pod that meets its resource requirements. This "sequential scheduling" strategy performs well when handling stateless, loosely coupled microservices. However, when faced with tightly coupled AI/HPC (High Performance Computing) workloads, it triggers two fatal problems: Deadlock and Resource Fragmentation.

5.1.1 Deadlock: The "Mahjong Tiles" That Can Never Be Completed

The deadlock problem primarily occurs in distributed training scenarios. Consider a PyTorch DistributedDataParallel task requiring 8 GPUs for data parallelism. This task typically consists of 8 Pods, each requesting 1 GPU. They need to synchronize via collective communication (e.g., All-Reduce), and training cannot begin until all are successfully started. (The following is a synthetic teaching example.)

Now, let us see how the native scheduler handles this task:

1. Job Submission: The user submits 8 Pods at once (let us call them train-job-pod-0 through train-job-pod-7).
2. Sequential Scheduling: kube-scheduler picks train-job-pod-0 from the pending queue, finds a node-A with an available GPU, and schedules it successfully. It then picks pod-1 and schedules it to node-B... This process continues.
3. Disaster Strikes: Assume the cluster has only 10 free GPUs in total. After the scheduler successfully places the first 5 Pods, they occupy 5 GPUs. Now, 5 free GPUs remain in the cluster. Unfortunately, another user submits an inference task requesting 5 GPUs. The scheduler "fairly" allocates these 5 cards to the new task. (What follows is a synthetic teaching scenario illustrating the deadlock mechanism.)
4. Deadlock Forms: Now, our training task has 5 Pods that have started and occupy GPUs, but they cannot work because they are anxiously waiting for their 3 sibling Pods. However, there are no more free GPUs in the cluster! The remaining 3 Pods (pod-5 to pod-7) will remain in Pending state forever.
5. Resource Waste: Worse still, the 5 Pods that have already started, unable to find their companions, just "spin in place," wasting 5 expensive GPUs without producing any computational value. The cluster's resources are "locked up" by partially occupied tasks, and new tasks cannot be scheduled either.

This scenario is like playing mahjong: you need to collect the "East," "South," "West," and "North" tiles to win. You already have "East," "South," and "West," but "North" was taken by someone else who has no intention of discarding it. The three tiles in your hand become useless, occupying space and preventing you from drawing new tiles.

The Native Scheduler's Fundamental Flaw: It lacks the "All-or-Nothing" atomic semantics. It does not know that these 8 Pods constitute an indivisible whole (a Job) that must be scheduled as a single unit.

5.1.2 Resource Fragmentation: Expensive "Islands"

Even without considering distributed tasks, the native scheduler can cause severe resource fragmentation when handling a large number of single-GPU tasks.

K8s's default scheduling strategy includes LeastAllocated (or similar BalancedResourceAllocation), which tends to distribute Pods evenly across all nodes for load balancing. This makes sense for web services, as it spreads risk. But for GPU clusters, this is a disaster.

Scenario: Assume we have 2 nodes, each with 8 GPUs (16 total). Now, we need to schedule 8 single-GPU training tasks, each requesting 1 GPU.

Native Scheduler Behavior:

• The first Pod arrives and is scheduled to node-A.
• For "balance," the scheduler tends to place the second Pod on the less loaded node-B.
• The third Pod goes to node-A...
• The final result is likely: node-A runs 4 Pods, and node-B runs 4 Pods. Each node has 4 GPUs occupied and 4 free.

Consequences of Fragmentation:

At this point, the cluster appears "load-balanced," but resource utilization is very low. If a large distributed training task requiring 8 GPUs arrives, the scheduler finds that no single node can meet its requirements! Even though there are 8 free GPUs in the cluster, they are scattered across two nodes, forming "resource fragments" that cannot be used by large tasks.

This is like a cinema selling tickets: a family of 8 wants to sit together, but the booking system, trying to fill every row, scatters empty seats all over the place, so the family can never find 8 consecutive seats.

The Native Scheduler's Fundamental Flaw: It lacks a "packing (Binpack)" rather than "scattering (Spread)" scheduling awareness. For expensive GPU resources, our goal should be to concentrate tasks on as few nodes as possible, preserving complete, contiguous "large chunks" of resources for future large tasks.

Summary:

The failure of the K8s native scheduler in AI/HPC scenarios stems from its design origin: it was built for stateless, loosely coupled, general-purpose applications. It lacks awareness of the "Job" concept as a whole and lacks the special consideration of "packing rather than scattering" expensive resources. To address these issues, we must introduce "advanced schedulers" designed specifically for batch processing and high-performance computing.

5.2 Advanced Scheduler in Practice: Installation, Configuration, and Strategy Details for Volcano/Yunikorn

To compensate for the shortcomings of the native scheduler, the cloud-native community has produced two excellent schedulers designed for batch processing systems: Volcano and Apache Yunikorn. Both run as "secondary schedulers" in K8s, specializing in handling complex workloads like AI and HPC.

5.2.1 How Schedulers Work: Competition and Cooperation

Multiple schedulers can run simultaneously in a K8s cluster. Pods can specify which scheduler should handle them via the spec.schedulerName field.

• Default Scheduler (default-scheduler): If a Pod does not specify schedulerName, it is handled by the default scheduler.
• Secondary Scheduler (e.g., volcano or yunikorn): We can configure the Pod templates for AI tasks to all specify schedulerName: volcano. This way, these Pods are managed by Volcano, while other ordinary applications in the cluster (like Nginx) remain managed by the default scheduler, without interference.

5.2.2 Volcano: CNCF's First Batch Scheduling System

Volcano is an open-source project contributed by Huawei to the CNCF (Cloud Native Computing Foundation). It is the first and currently most mature cloud-native batch processing system in the community. It is not just a scheduler but a complete job management system.

Core Concepts:

Job & PodGroup: Volcano introduces the Job CRD (Custom Resource Definition), representing a complete batch processing job. A Job can contain multiple tasks (Tasks), each corresponding to a Pod template. At a lower level, Volcano logically binds all Pods in a Job into a PodGroup. The PodGroup is the basic unit for Volcano's atomic scheduling.

Queue: Volcano introduces the concept of a Queue. All jobs must be submitted to a queue. Administrators can set different resource quotas, priorities, and scheduling policies for different queues. This provides a powerful mechanism for multi-tenant resource isolation and fair sharing. For example, a high-priority, high-quota queue could be created for the "core algorithm team" and a low-priority queue for the "intern team."

Installation (using Helm):

helm repo add volcano-sh https://volcano-sh.github.io/charts
helm repo update
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace

After installation, you will see Pods for core components like volcano-scheduler and volcano-controller-manager running in the volcano-system namespace.

How to Submit a Job Using Volcano:

You need to write a YAML file for a VolcanoJob.

Example: A PyTorch distributed training VolcanoJob requiring 4 Pods

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
    name: pytorch-dist-job
spec:
    schedulerName: volcano # Specify the scheduler
    minAvailable: 4 # This is key! Implements Gang Scheduling
    queue: default # Submit to the default queue
    tasks:
    - name: worker
        replicas: 4 # Total of 4 replicas needed
        template:
        spec:
            containers:
            - name: pytorch
                image: my-pytorch-app:1.0
                resources:
                limits:
                    nvidia.com/gpu: 1
            restartPolicy: OnFailure

minAvailable: 4: This is the mechanism behind Volcano's Gang Scheduling. It tells Volcano: this job needs the resources for at least 4 Pods to be satisfied simultaneously before you can start scheduling; otherwise, do not touch any of them -- let them wait in the queue. This dissolves the deadlock problem at the scheduling-semantics level.

Volcano's Core Advantages:

• Rich Scheduling Strategies: Besides Gang Scheduling, Volcano supports multiple plugin-based scheduling algorithms, such as Binpack, DRF (Dominant Resource Fairness), Priority, and others.
• Powerful Queue Management: Queue-based resource isolation and preemption mechanisms are very mature.
• Integration with Mainstream Frameworks: Deeply integrated with Spark, Flink, TensorFlow, PyTorch, and others, providing Operators to simplify job submission.

5.2.3 Apache Yunikorn: Designed for Large Scale and Mixed Workloads

Yunikorn is a cloud-native scheduler driven by companies like Cloudera and Microsoft, originating from the experience of the Hadoop YARN scheduler. Its design emphasizes extreme scalability (tens of thousands of nodes), high performance, and resource fairness.

Core Concepts:

• Hierarchical Queues: This is Yunikorn's biggest feature. Its queues can form a tree structure, like a filesystem directory. Resource quotas can be inherited from parent queues or overridden on child queues. This provides immense flexibility for large organizations to perform fine-grained resource partitioning and management. For example: root.research.team-a and root.production.online-service.
• Application: Yunikorn automatically recognizes a group of related Pods (e.g., all Executors in a Spark job) as an "Application," which is also its basic scheduling unit.

Installation: Yunikorn can also be installed via its official Helm Chart, similar to Volcano.

How to Use Yunikorn:

Yunikorn emphasizes "seamless" integration. It automatically identifies Pods that are associated via labels and schedules them as an application. You still submit standard K8s Pods, but you need to add label and annotation information that Yunikorn can recognize.

apiVersion: v1
kind: Pod
metadata:
    name: my-app-pod-1
    labels:
    applicationId: "app-0001" # Key! Marks as belonging to the same application
    queue: "root.my-queue" # Specify the queue
spec:
    schedulerName: yunikorn # Specify the scheduler
    # ...

Yunikorn's Core Advantages:

• Excellent Scheduling Performance: Specifically optimized for large-scale clusters, with very high scheduling throughput.
• Flexible Hierarchical Queues: A killer feature for enterprises with complex organizational structures and resource management needs.
• Excellent Resource Fairness Algorithms: Built-in Fairness and DRF algorithms effectively balance resource allocation across multiple users and tasks.

5.2.4 Volcano vs. Yunikorn: How to Choose?

Advice for AI Infra Engineers:

• If your team primarily runs tightly coupled batch processing jobs like deep learning and HPC, and wants out-of-the-box Gang Scheduling and job lifecycle management, Volcano is the more direct and mature choice.
• If you manage a hyperscale, multi-tenant, mixed-workload environment (with AI training, big data processing, and online services), requiring very fine-grained hierarchical resource budgeting and fairness guarantees, then Yunikorn's powerful hierarchical queues and scheduling performance will be more attractive.

In many real-world cases, both can effectively solve the problems of the native scheduler. The choice depends more on your team's background, technology stack preferences, and organizational management complexity.

5.3 Key Scheduling Strategies: Gang Scheduling, Binpack, Topology-Aware Scheduling

Regardless of whether you choose Volcano or Yunikorn, their powerful scheduling capabilities stem from a series of ingenious scheduling algorithms behind them (usually implemented as "plugins" in the scheduler). Understanding these core strategies is what allows you to truly "tune" your scheduler.

5.3.1 Gang Scheduling: The Silver Bullet for Solving Deadlocks

Strategy Goal: All-or-Nothing. Ensure that all members (Pods) of a gang (job) can be scheduled simultaneously. If resources are insufficient to satisfy all members at once, the entire job remains waiting and does not occupy any resources.

Implementation Principle (using Volcano as an example):

1. Queuing and Validation: When a VolcanoJob is submitted to a queue, the Volcano Controller creates a PodGroup object for it, recording minAvailable (the minimum number of replicas required). Before starting to schedule the job, the scheduler performs a pre-check: it simulates scheduling to see if the cluster's available resources are sufficient to meet the requirements of minAvailable Pods.
2. Resource Reservation (Gating): If the pre-check passes, Volcano enters the "Gating" stage. It temporarily "locks" or "reserves" this portion of resources, and only then starts creating and scheduling Pods one by one. Because the resources are already reserved, there is no risk of them being taken by others mid-scheduling.
3. Timeout and Backoff: If the pre-check fails, or if the waiting time for resources exceeds a certain timeout, the entire job returns to the queue to continue waiting, and may trigger preemption of other lower-priority jobs to release resources.

Value: Eliminates at the scheduling-semantics level the deadlock problem caused by some Pods failing to start in distributed training. This is a "standard feature" and core value of batch processing schedulers.

5.3.2 Binpack: The Cure for Resource Fragmentation

Strategy Goal: Pack rather than spread. As much as possible, concentrate Pods onto a few nodes until those nodes' resources are exhausted, before using new nodes.

Implementation Principle:

• The Binpack strategy gives higher scores to nodes that already have high resource utilization when scoring Pod placement.
• For example, a simple Binpack scoring function could be: score = (gpu_used / gpu_total) * 10.
• node-A has 6/8 GPUs in use, score (6/8)*10 = 7.5.
• node-B has 2/8 GPUs in use, score (2/8)*10 = 2.5.
• The scheduler prefers node-A with the higher score.
• Thus, node-A gets filled up quickly, while node-B remains completely free, becoming a "whole block of resources" available for future large tasks. This significantly reduces resource fragmentation.

Value: Dramatically improves the overall resource utilization of the cluster. For large jobs that require entire machines or multiple machines, the Binpack strategy is essential. It represents a trade-off with the Spread strategy (which pursues load balancing) that requires careful consideration in AI clusters. Typically, for expensive, non-divisible resources like GPUs, Binpack is the superior choice.

5.3.3 Topology-Aware Scheduling: Pursuing Extreme Communication Performance

Strategy Goal: Schedule Pods that need to communicate frequently as physically close to each other as possible, to reduce network latency and improve distributed training performance.

Levels of "Closeness":

• Level 0: Within the Same Node. This is the ideal scenario. If an 8-card task can be scheduled onto a single 8-GPU server, communication between the cards will primarily use the fastest NVLink/HCCS, yielding the best performance.
• Level 1: Within the Same Rack. If the task spans nodes, scheduling them onto different nodes within the same rack means their communication only needs to pass through one layer of aggregation switch (Top-of-Rack Switch), with lower latency.
• Level 2: Different Racks. If Pods are scheduled to different racks, communication must traverse multiple layers of switches, increasing latency and congestion risk.

Implementation Principle:

1. Topology Information Collection: The administrator needs to pre-label K8s Node objects with topology labels. These labels are typically maintained by automation scripts or a CMDB system.

# Label nodes with zone and rack information
kubectl label node node-01 topology.kubernetes.io/zone=zone-a topology.kubernetes.io/rack=rack-01
kubectl label node node-02 topology.kubernetes.io/zone=zone-a topology.kubernetes.io/rack=rack-01
kubectl label node node-03 topology.kubernetes.io/zone=zone-b topology.kubernetes.io/rack=rack-03

2. Scheduler Scoring: The topology-aware scheduling plugin reads these labels. When scheduling a gang job, it tends to place all Pods of that gang onto node groups sharing the same topology.kubernetes.io/rack label. This can be a filtering condition (hard constraint) or a scoring item (soft constraint).
3. Volcano's Implementation: Volcano's Affinity and Topology plugins support scheduling based on these labels. You can define a PodGroup requiring all its members to have a certain affinity.

Value: For large-scale, communication-intensive AI training tasks, the performance improvement from topology-aware scheduling is tangible. It can improve the cluster's linear scalability by several or even over ten percentage points, translating into substantial savings in training time and money. This is a key step from "making it work" to "making it fast," and is an optimization capability that top-tier AI Infra teams must possess.

Summary:

In this chapter, we completed an evolution from "usable" to "effective." By introducing advanced schedulers like Volcano or Yunikorn, and deeply understanding the core strategies behind them -- Gang Scheduling, Binpack, and Topology-Aware Scheduling -- we built a truly intelligent and efficient AI resource scheduling system. It is no longer a "fool" blindly allocating resources, but an "expert" skilled in trade-offs and planning. At this point, the "foundation" of our cloud-native AI platform is fully built. Next, we will enter the core third part of the book, shifting our focus from the platform layer to the application layer, confronting the full-process operations challenges of LLM training and inference.