In Chapter 7, we explored "board-level" compute scheduling technologies, focusing on how to share a single physical GPU card among multiple virtual machines or containers. We learned about various technologies such as GPU pass-through, vGPU, MIG, and container-based device plugins. These can be seen as the "first level" of GPU virtualization, mostly relying on hardware vendors' native support, each with its own emphasis on isolation, performance, and implementation, but sharing the common goal of transforming physical GPUs into "GPU resource blocks" recognizable and allocable by upper-layer scheduling systems.
However, with the increasing diversification of AI application scenarios, especially the flourishing development of cloud-native AI platforms, simply being satisfied with this "resource block" style of allocation can no longer meet the industry's pursuit of ultimate elasticity, utilization, and flexibility. For example:
A user might only need 100 MB of memory and minimal compute power to run a TensorBoard service -- even the smallest MIG instance (like 5 GB memory) would be a huge waste.
An inference task might require substantial compute power during peak hours but be nearly idle during off-peak hours. Can we achieve "on-demand scaling" of compute power instead of static binding?
Can we truly "pool" the memory and compute power of all GPUs in the cluster into a unified, huge virtual GPU resource pool, then allocate it on-demand and precisely (e.g., in units of 1 MB memory, 1% compute) to any container, just like CPU and memory?
To achieve these more ambitious goals, we need to enter the "second level" of GPU virtualization. This chapter will focus on an in-depth exploration of various GPU virtualization scheduling solutions. These are no longer just about resource allocation; they virtualize, pool, and finely manage GPU compute and memory resources through deeper technical means. These solutions are often more "invasive," intercepting and redirecting CUDA APIs, drivers, and even hardware instructions to create a "convincing" virtual GPU environment.
We will comprehensively examine the ecosystem from three dimensions:
- NVIDIA's Official GPU Virtualization Scheduling Solutions: We will review and delve into NVIDIA's own technical system, from the basic API Remoting to enterprise-grade GRID vGPU to hardware-level MIG, understanding its official layered answers to different virtualization needs.
- Other Hardware Vendors' GPU Virtualization Scheduling Solutions: We will briefly introduce the virtualization approaches and technical implementations of AMD and Intel, such as SR-IOV and GVT-g, to form a broader industry view.
- Cloud Vendor and Open Source Community Container-Based GPU Virtualization Scheduling Solutions: This is the focus of this chapter. We will deeply analyze the innovative technologies exemplified by Alibaba Cloud cGPU, Tencent Cloud qGPU, and open-source community solutions that shine in cloud-native environments. Most of these solutions are container-based, using API hooking and other techniques to achieve ultra-fine-grained sharing and pooling of GPU compute and memory resources, representing the cutting edge of improving GPU cluster resource utilization.
Through the study of this chapter, you will master a complete GPU virtualization technology map. From hardware vendors' "orthodox" solutions to cloud vendors' and open-source communities' "clever tricks," you will understand their core principles, technical trade-offs, and most suitable application scenarios. This will enable you to make the highest-level architectural decisions regarding resource utilization, isolation, performance, and cost when building next-generation AI platforms.
8.1 NVIDIA's GPU Virtualization Scheduling Solutions
As the absolute leader in the GPU domain, NVIDIA's own GPU virtualization technology system has evolved over many years, forming a set of layered solutions covering different scenarios. Understanding this "official" system is the foundation and reference for understanding all other virtualization solutions.
8.1.1 Foundation Layer: API Remoting and vCUDA
Before heavyweight solutions like vGPU and MIG emerged, academia and early industry were already exploring software-based GPU sharing. The most core and basic idea is API Remoting.
Core Principle:
- Client: On a client machine (VM or container) without a physical GPU, provide a "fake" CUDA runtime library (shim library) with the exact same API interface as the native
libcuda.so. - Server: On a server with a physical GPU, run a service daemon.
- API Interception and Forwarding: When the client application calls a CUDA API (e.g.,
cudaMalloc), this call does not execute locally but is intercepted by the "fake" library. - The intercepted library serializes the API call's name, parameters, etc., and forwards them over a network (like TCP/IP) to the remote server daemon.
- Remote Execution and Result Return: Upon receiving the request, the server daemon deserializes the original API call and actually executes it on the local physical GPU.
- After execution, if the API has a return value or needs to return data via pointers, the server daemon serializes the result again and sends it back over the network to the client's "fake" library, which returns it to the application.
vCUDA Project: vCUDA is an early, well-known open-source academic project based on the API Remoting idea. It fully implemented the above flow, allowing multiple VMs without GPUs to share GPU resources on a single physical server.
Advantages:
Simple and Intuitive Architecture: Conceptually clear and easy to understand.
Breaking Physical Boundaries: For the first time, achieved cross-node, networked sharing of GPU resources, theoretically allowing pooled GPUs across an entire data center.
Disadvantages:
Severe Performance Bottleneck:
Network Latency: Every API call requires a network round trip. For applications that frequently call a large number of small APIs, the performance overhead is catastrophic.
Data Transfer: When API calls involve large data transfers (like cudaMemcpy), all data must be copied through the CPU and network, completely losing the bandwidth advantages of PCIe and NVLink.
High Kernel Launch Overhead: Launching a compute kernel itself becomes a network operation costing hundreds of microseconds or even milliseconds.
Compatibility Issues: Requires rewriting and adapting the "fake" library for each new CUDA driver and API version, with extremely high maintenance costs.
Incomplete Functionality: It is difficult to fully implement all complex CUDA functionalities, especially those involving low-level hardware interaction and pointer operations.
Current Status and Significance: Pure API Remoting solutions, due to their inherent performance flaws, are rarely directly used for high-performance AI training today. However, their core idea -- API interception and forwarding -- has been adopted and developed by many subsequent, more advanced virtualization solutions. It is the intellectual source for understanding technologies like cGPU.
8.1.2 Enterprise Standard: NVIDIA GRID vGPU
We already introduced NVIDIA vGPU's principles in Chapter 7. Here, we place it within NVIDIA's overall solution to emphasize its positioning.
Positioning: GRID vGPU is NVIDIA's officially promoted, commercial GPU virtualization solution for enterprise virtualization environments (mainly VMware vSphere, Citrix Hypervisor, etc.). Its product brand has evolved several times and is now mainly integrated within software suites like NVIDIA AI Enterprise (NVAIE) and NVIDIA RTX Virtual Workstation (vWS).
Core Technology Review:
Mediated Pass-through: Unlike API Remoting, vGPU's API forwarding occurs between the VM and the Hypervisor through an efficient, dedicated VMM channel, not a general-purpose network.
vGPU Manager: As the "general manager" running in the Hypervisor, it handles the creation, scheduling, and isolation of all vGPU instances.
Time-Slicing Scheduling: At the hardware level, time-slicing technology allows multiple vGPU instances to take turns using the physical GPU's compute engine.
Memory Isolation: The vGPU Manager allocates a fixed, protected region of physical memory for each vGPU instance, ensuring strong memory isolation.
Role in the NVIDIA System:
The "Orthodox" Solution for Virtualized Environments: It is the only solution officially fully supported by NVIDIA, providing complete functionality, performance, and enterprise-level services on mainstream virtualization platforms like VMware.
Covering Graphics and Compute: vGPU not only supports CUDA computing but also perfectly supports graphics APIs like OpenGL and DirectX, making it widely used in cloud desktop (VDI) and cloud gaming scenarios.
Software-Defined Flexibility: Through the vGPU Manager and different vGPU profiles, administrators can flexibly allocate different amounts of memory to different VMs and apply different scheduling strategies.
8.1.3 Hardware-Level Virtualization: NVIDIA MIG
MIG was also introduced in Chapter 7. It represents NVIDIA's latest approach to solving GPU sharing at the hardware level.
Positioning: MIG is a multi-tenant sharing solution providing strong isolation for bare-metal and containerized environments. It is particularly suitable for scenarios where multiple mutually untrusted AI tasks requiring performance guarantees run on a single physical node, such as public cloud PaaS/CaaS platforms.
Core Technology Review:
Spatial Slicing: MIG statically and spatially partitions the GPU's physical resources (SMs, L2 cache, memory controllers, etc.) at the hardware level.
GPU Instance (GI): Each partitioned unit is a GI. Each GI has its own independent, interference-free hardware resource path.
Compute Instance (CI): Within a GI, further CIs can be created. CIs share the GI's memory but have their own independent compute engine states, suitable for scenarios requiring isolated execution contexts but sharing data.
MIG vs. vGPU Comparison:
Isolation: MIG's isolation is at the hardware level, stronger and more thorough than vGPU's software arbitration isolation. A fault in one GI normally does not affect another GI; however, this is fault and performance isolation, not an absolute security boundary against malicious side-channel attacks.
Performance: MIG has no API forwarding overhead. Each GI's performance is predictable and guaranteed because it exclusively owns a portion of physical hardware. vGPU's performance is shared and competitive; a vGPU's actual performance depends on how many other vGPUs are working simultaneously.
Flexibility: vGPU sharing is dynamic (time-slicing). When a vGPU is idle, other vGPUs can use the full compute power. MIG partitioning is static; even if a GI is idle, its hardware resources cannot be used by other GIs.
Memory: vGPU memory allocation is more flexible (determined by profiles). MIG memory allocation is tied to hardware slices, with fixed specifications.
Application Environment: vGPU mainly targets VM environments. MIG is more suitable for bare-metal and container environments, integrating well with Kubernetes' Device Plugin.
NVIDIA Solution Summary:
NVIDIA provides a full spectrum of solutions, from software to hardware, from flexible sharing to strong isolation:
MIG offers the strongest isolation and performance guarantees but slightly less flexibility. It is the ideal choice for multi-tenant container platforms.
vGPU offers the most balanced flexibility, isolation, and functional completeness. It is the preferred commercial solution for enterprise virtualized environments (VM).
MPS (Multi-Process Service) is a lightweight time-slicing solution suitable for sharing a card among multiple small tasks (like inference services) in a single-user, trusted environment to improve throughput.
The idea of API Remoting, as a foundational technology, has been developed by the open-source community and cloud vendors, giving rise to more innovative virtualization solutions.
8.2 GPU Virtualization Solutions from Other Hardware Vendors
Although NVIDIA dominates the data center GPU market, understanding AMD and Intel's approaches provides a more complete picture. Their solutions mostly revolve around standardized I/O virtualization technologies.
8.2.1 AMD's SR-IOV Solution
AMD is a major proponent of SR-IOV technology in GPU virtualization. Its MxGPU technology is implemented based on SR-IOV.
Core Principle:
AMD's data center GPUs (like the Instinct MI series) support SR-IOV at the hardware level.
By loading the Physical Function (PF) driver in the Hypervisor, a single physical GPU can be partitioned into multiple Virtual Functions (VFs) at the hardware level. For example, an MI100 can be partitioned into up to 8 VFs.
Each VF has its own independent scheduling queues, memory page tables, and interrupts, appearing as an independent GPU device from the hardware perspective.
These VFs can be directly passed through to different VMs.
VMs load AMD's VF driver internally and use the VF like a physical GPU.
Comparison with NVIDIA vGPU:
Implementation: AMD SR-IOV is a hardware pass-through model with minimal performance overhead. NVIDIA vGPU is an API forwarding model with software arbitration overhead.
Isolation: Both provide hardware-assisted strong isolation.
Flexibility: SR-IOV's VF partitioning specifications are fixed and inflexible. vGPU's profiles and time-slicing scheduling offer higher flexibility.
Ecosystem: NVIDIA vGPU's ecosystem is more mature, with deeper integration with mainstream virtualization platforms like VMware and Citrix, and more comprehensive commercial support.
8.2.2 Intel's GVT-g Solution
Intel, in promoting its integrated graphics and dedicated data center GPUs (like Ponte Vecchio, Gaudi), has also introduced its own GPU virtualization technology, with GVT-g (Graphics Virtualization Technology -g) being its representative work.
Core Principle: GVT-g uses an API forwarding/mediation model similar to NVIDIA vGPU, but it is completely open source and deeply integrated into the Linux kernel (KVMGT) and QEMU.
No Guest Driver Needed: Unlike vGPU, a clever aspect of GVT-g is that it does not require any special driver to be installed inside the VM. The VM loads a standard, open-source Intel i915 graphics driver, completely unaware it is running in a virtualized environment.
Hypervisor Arbitration: When the Guest driver submits graphics or compute commands, the KVMGT module in the Hypervisor intercepts the operations that write to the underlying hardware registers, then safely submits these commands to the physical GPU on behalf of the VM.
Memory Virtualization: GVT-g virtualizes the GPU's address space (GGTT) for each VM using Shadow Page Table technology, achieving memory isolation.
Advantages:
Open Source and Free: GVT-g is completely open source, requiring no commercial license.
Good Compatibility: No special Guest driver needed, simplifying deployment.
Advanced Technology: Its design concepts (like no Guest driver, shadow page tables) are very elegant.
Disadvantages:
Ecosystem and Maturity: GVT-g mainly revolves around Intel's own GPU products. Its ecosystem and market recognition in large-scale AI computing in data centers still lag far behind NVIDIA.
Performance: As a software arbitration solution, it also has some performance overhead.
8.3 Cloud Vendor and Open Source Community Container-Based GPU Virtualization Solutions
Under the cloud-native wave, finding a GPU sharing solution for containers that is more flexible than MIG, has better isolation than time-slicing, and costs less than vGPU has become the "Holy Grail" pursued by major cloud vendors and the open-source community. The core idea of these solutions can mostly be traced back to API Remoting, but they have made extensive innovations and optimizations in implementation.
Core Idea: Localized API Hooking and Resource Management
The general architectural pattern of these solutions is as follows:
CUDA API Hooking
They provide a custom libcuda.so dynamic link library. When the user's container starts, through the LD_PRELOAD environment variable, the application is forced to load this custom library instead of the system's native CUDA library.
This custom library "hooks" all CUDA API calls made by the application.
Resource Management and Scheduling Daemon
A resident management daemon runs on each GPU node.
This daemon manages the real resources (compute, memory) of all physical GPUs on this node.
It maintains a "ledger" of virtual GPU resources allocated to each container (e.g., Container A has 2 GB memory and 20% compute).
Local IPC Communication
Hooked API calls are not forwarded over the network but are sent to the local management daemon via efficient local IPC (Inter-Process Communication), such as Unix Domain Sockets. This avoids the huge network overhead of API Remoting.
On-Demand Simulation and Resource Limiting
Memory Management: When the daemon receives a cudaMalloc request, it checks if the container's memory quota is sufficient. If so, it allocates memory on the real physical GPU and establishes a mapping from "virtual memory address" to "real memory address." It "lies" to the container about the total GPU memory, reporting it as its allocated quota.
Compute Management: For compute kernel launches, the daemon controls the kernel's execution opportunities on the physical GPU based on the container's compute quota. This can be achieved in several ways, such as:
Limiting kernel concurrency: Restricting the maximum number of thread blocks this container can have running on the GPU simultaneously.
Dynamically adjusting SM frequency or power limits (requires hardware support).
More refined time-slicing: Combining NVIDIA MPS or its own scheduler to precisely control execution time.
8.3.1 Alibaba Cloud cGPU
cGPU (container GPU) is the GPU sharing virtualization solution launched by Alibaba Cloud Container Service.
Core Features:
Decoupled Compute and Memory: Users can independently request compute power and memory. For example, a container with only 1 GB memory but 50% compute power, or one with 10 GB memory but only 10% compute.
Memory Isolation: Through API hooking, cGPU virtualizes an independent memory space for each container. One container cannot access another container's memory.
Compute Isolation: By controlling CUDA kernel execution (similar to time-slicing), it achieves compute isolation and limiting.
Deep Integration with Kubernetes: cGPU provides its own Device Plugin. Users can declare resources directly in Pod YAML as aliyun.com/gpu-mem: 1024 (in MB) and aliyun.com/gpu-core: 50 (in %). The scheduler dispatches based on the node's cGPU resource availability.
8.3.2 Tencent Cloud qGPU
qGPU (GPU TKE-Ving) is the GPU virtualization solution launched by Tencent Cloud Container Service (TKE). Its ideas are similar to cGPU, but with differences in implementation details and commercialization.
Core Features:
Similarly achieves fine-grained partitioning and isolation of compute and memory.
Emphasizes QoS guarantees, providing differentiated service quality for tasks of different priorities.
Deeply integrated with Tencent Cloud's own monitoring, billing, and operations systems.
8.3.3 Open Source Community Solutions: TKE vCUDA + GPU Manager
In addition to closed-source commercial solutions, the open-source community has also seen many similar projects, such as Tencent Cloud's early open-sourced TKE GPU-Manager.
TKE vCUDA: A libcuda.so library implementing local API hooking.
GPU Manager: A scheduling and management component integrated with Kubernetes.
It allows users to declare GPU requirements in Pod Annotations (e.g., tke.cloud.tencent.com/gpu-core-percentage: 30, tke.cloud.tencent.com/gpu-mem-percentage: 30).
A custom scheduler extender decides which node the Pod should be scheduled to based on these Annotations and the node's GPU resource usage.
The GPU Manager on the node sets environment variables like LD_PRELOAD for the started container, enabling vCUDA's API hooking.
8.3.4 Commonality, Advantages, and Challenges
Commonality: They all adopt the core architectural pattern of "application-transparent, local hooking, centralized management and control." From the perspective of the user's AI application code, it is completely unaware it is running in a virtualized GPU environment.
Advantages:
Ultimate Flexibility and Utilization: Achieves ultra-fine-grained partitioning and oversubscription of GPU resources. In scenarios with many shared tasks and fragmented workloads it can markedly raise GPU utilization (public cloud-vendor case studies report varying magnitudes, commonly claiming an increase from the usual two-to-three tenths to above sixty percent), thereby reducing unit computing cost; the actual gain depends on workload mix and should be verified against one's own cluster statistics.
Decoupled Compute and Memory: Accommodates various unusual resource requirements, adapting to all scenarios from development, debugging, inference, to small-scale training.
Easy Integration: Seamlessly integrates with the cloud-native ecosystem (Kubernetes, Docker), providing a good user experience.
Challenges and Trade-offs:
Performance Overhead: API hooking and IPC communication, though much faster than network forwarding, still introduce some performance overhead, especially for applications that frequently call a large number of small APIs. For large-scale distributed training requiring ultimate performance, the performance may not match MIG or physical cards.
Compatibility and Maintenance Cost: The biggest challenge is keeping up with NVIDIA driver and CUDA version updates. Whenever NVIDIA releases a new driver, these solutions' libcuda.so hooking libraries may need adaptation and rigorous testing to ensure all CUDA APIs are correctly simulated and handled. This is a huge, continuous engineering investment.
Isolation Strength: This software-level isolation, in theory, is not as strong or secure as hardware/Hypervisor-level isolation like MIG or vGPU. Although it can prevent most routine errors, potential risks still exist under the strictest security requirements.
Functional Completeness: Some very low-level or undocumented CUDA features may be difficult to perfectly simulate, causing some special applications to fail.
8.4 Chapter Summary
In this chapter, we conducted a panoramic and in-depth examination of GPU virtualization scheduling solutions, elevating our understanding of GPU resource sharing from "board-level allocation" in Chapter 7 to the higher level of "fine-grained virtualization."
We sorted through NVIDIA's official virtualization technology "family bucket," clarifying their respective positions:
MIG, with its unparalleled hardware-level strong isolation and predictable performance, becomes the cornerstone of security and performance in cloud-native multi-tenant scenarios.
GRID vGPU, with its mature ecosystem in VM environments, flexible time-slicing scheduling, and complete functionality, holds the top spot in the enterprise virtualization market.
API Remoting, as a foundational idea, laid the theoretical groundwork for many subsequent innovative solutions.
We also horizontally compared solutions from other hardware vendors, such as AMD's SR-IOV and Intel's GVT-g. We saw that they represent two different technical paths -- hardware pass-through and open-source arbitration -- and although their ecosystems lag behind NVIDIA's, their technical ideas are equally valuable for reference.
The highlight of this chapter was our deep analysis of the container GPU virtualization solutions represented by Alibaba Cloud cGPU and Tencent Cloud qGPU, which emerged from the cloud-native community. We revealed their common, clever core architecture: through localized CUDA API hooking and a centralized resource management daemon, they successfully achieve ultra-fine-grained partitioning, isolation, and pooling of GPU compute and memory while keeping the application unaware. These solutions, with their extreme flexibility and resource utilization, perfectly align with the cloud-native era's pursuit of elasticity and cost-effectiveness, representing the most active and innovative development direction in current GPU virtualization. However, they also face ongoing challenges in performance overhead, compatibility maintenance, and isolation strength.
Ultimately, we conclude: There is no "silver bullet" GPU virtualization solution. This is a decision space full of trade-offs. When choosing a solution, an architect must act like an experienced chef, carefully blending various "seasonings" (technical solutions) based on the "ingredients" (business scenario) and desired "flavor" (core requirements):
If security isolation is the top priority, choose MIG or vGPU.
If extreme resource utilization and cost-effectiveness are the primary goals, choose solutions like cGPU/qGPU.
If lossless bare-metal performance is required and exclusivity is acceptable, choose GPU pass-through or simple exclusive container scheduling.
A profound understanding of this GPU virtualization technology map will enable us to build a computing foundation for our AI platform that is both powerful and economical, both stable and flexible, thereby gaining critical "cost advantage" and "agility advantage" in the intense AI arms race.