祝融说。

In Chapter 6, we focused on building the physical network of the GPU cluster, like planning a modern city with highways (compute network), arterial roads (storage network), and local streets (business network). This physical network is the solid foundation for all data flow. However, in a real multi-user, multi-task, multi-tenant cloud environment, merely having a shared physical network is far from enough.

Imagine if we let all users' "vehicles" (data packets) drive freely on the same huge, flat physical transportation network. What problems would arise?

Address Conflicts: Different users or applications may want to use the same IP address ranges (e.g., they all favor using 192.168.1.0/24), leading to disastrous address conflicts.

Lack of Security Isolation: Any user could easily "see" or even "attack" another user's data packets, with no privacy or security whatsoever.

Broadcast Storms: One user's broadcast or multicast traffic would spread across the entire physical network, affecting all users.

Inflexible Routing Policies: The physical network's routing policy is uniform, unable to provide customized, flexible routing rules for different users or applications.

To solve these problems, we must build a virtualized network layer on top of the physical network. This chapter will delve into the design and implementation of GPU cluster network virtualization. Our core goal is to learn how to virtualize a shared physical network infrastructure, through software-defined technologies, into hundreds or thousands of independent, logically completely isolated private networks. Each private network is like a tenant-exclusive "city within a city," with its own independent address space, routing table, security policies, and network services.

We will start with the cornerstone technology of network virtualization -- SDN-based VPC -- understanding how it achieves network resource pooling and isolation. Next, we will focus on the indispensable component of machine learning networks -- cloud load balancing -- exploring how it provides high availability and scalability for AI inference services. Then, we will learn how to connect the virtual network securely with the outside world (such as user data centers, other VPCs, the public internet) through technologies like dedicated line access, peering connections, and VPC gateways. Finally, we will dive into the low-level details to analyze the core engine for implementing various advanced network functions -- the SDN NFV gateway -- and how it is deployed and accelerated.

Through the study of this chapter, you will master a complete set of cloud-native network virtualization architecture knowledge. You will no longer just see physical switches and routers; you will be able to understand and design advanced network services like VPCs, subnets, security groups, load balancers, and NAT gateways from a cloud architect's perspective. This will enable you to build a flexible and powerful virtual network infrastructure for a multi-tenant, secure, feature-rich GPU cloud platform.

9.1 SDN-Based VPC Technology: The Foundation of Network Virtualization

VPC (Virtual Private Cloud) is the core concept and de facto standard for network virtualization in modern cloud computing. It allows users to build a logically completely isolated private network environment within a shared public cloud infrastructure, fully controlled by the user. The underlying technology for implementing VPC is SDN (Software-Defined Networking).

9.1.1 Core Idea of SDN: Separation of Control and Forwarding

In traditional network devices (like switches, routers), the control plane and data plane are tightly coupled within the same device.

Control Plane: Responsible for "thinking" and "decision-making." It runs various routing protocols (like OSPF, BGP), learns network topology, computes routing tables, and generates forwarding tables.

Data Plane: Responsible for "execution." It uses dedicated ASIC chips to perform high-speed lookup and forwarding of data packets based on the forwarding tables generated by the control plane.

The core idea of SDN is to decouple these two planes:

1. Data Plane (Infrastructure Layer): The underlying physical switches are "simplified" into "dumb" devices only responsible for high-speed forwarding (sometimes called "white-box switches"). They no longer run complex routing protocols.
2. Control Plane (SDN Controller): A centralized SDN controller running on a server becomes the "brain" of the entire network. It has a global view of the network topology and is responsible for all route computation, policy formulation, and traffic engineering.
3. Southbound Interface: The SDN controller communicates with all underlying physical switches through a standard protocol (like OpenFlow, OVSDB). The controller pushes computed forwarding table rules (Flow Table) down to the switches.
4. Northbound Interface: The SDN controller provides RESTful APIs and other programming interfaces to upper-layer applications (like VPC management platforms, cloud orchestration systems), allowing the network to be managed and configured programmatically and automatically.

9.1.2 Overlay Networks: Building "Tunnels" for the Virtual World

Merely separating control and forwarding is not enough to achieve VPC isolation. SDN typically combines Overlay network technology to build virtual networks.

Underlay Network: Refers to the underlying physical network infrastructure (Spine-Leaf switches and routers). The goal of the Underlay network is simple: achieve IP packet reachability between physical devices.

Overlay Network: A virtual, logical network built on top of the Underlay network through tunneling technology.

Working Principle: When a VM (Tenant A) wants to send a data packet to another VM in the same VPC but on a different physical machine, the original data packet (inner packet) is first sent to a virtual switch (vSwitch) on the source host.

The vSwitch does not directly send this inner packet to the physical network. Instead, it encapsulates it as "cargo" in the data portion of a new IP packet (outer packet). The source IP of this outer packet is the source host's physical IP, and the destination IP is the destination host's physical IP. This encapsulation process is like establishing a "tunnel."

This outer packet is normally routed and forwarded in the Underlay physical network until it reaches the destination host.

The vSwitch on the destination host receives this outer packet, "decapsulates" it, extracts the original inner packet, and delivers it to the target VM.

Tunneling Protocols: There are many tunneling protocols for implementing Overlay networks, with VXLAN (Virtual eXtensible LAN) and Geneve being the most mainstream.

VXLAN: Encapsulates the original Ethernet frame in a UDP packet. It introduces a 24-bit VNI (VXLAN Network Identifier) field, which uniquely identifies a virtual network. All VXLAN packets with the same VNI belong to the same VPC. The 24-bit VNI theoretically supports up to 16 million independent virtual networks, far exceeding the limit of traditional VLANs (only 4096).

9.1.3 VPC Implementation Flow Based on SDN

Now, combining SDN and VXLAN, let us see how a typical VPC works:

1. User Creates a VPC: The user creates a VPC through the cloud platform's UI or API, defining a private IP address range (CIDR), e.g., 10.0.0.0/16.
2. Controller Assigns a VNI: The SDN controller, upon receiving this request, assigns a globally unique VNI to this new VPC, e.g., 1001.
3. User Creates Subnet and VM: The user creates a subnet (e.g., 10.0.1.0/24) within the VPC and launches a VM-A in that subnet.
4. Controller Dispatches Rules:
   1. The SDN controller now knows that VM-A's virtual IP is 10.0.1.10, its MAC address is mac-a, and the physical host it is on has IP 1.1.1.1.
   2. The controller distributes this information (the mac-a -> 1.1.1.1 mapping) to all vSwitches in the network.
5. VPC Internal Communication:
   1. When VM-A (10.0.1.10) in the same VPC wants to communicate with VM-B (10.0.2.20) on another physical host, VM-A sends a packet destined for 10.0.2.20.
   2. The vSwitch on VM-A's host intercepts this packet. It looks up the forwarding table dispatched by the SDN controller and finds that 10.0.2.20 corresponds to physical host 1.1.1.2, and they both belong to VNI 1001.
   3. The vSwitch encapsulates VM-A's original packet with VNI 1001 using VXLAN, then wraps it in a new IP packet with source IP 1.1.1.1 and destination IP 1.1.1.2.
   4. This encapsulated packet is forwarded in the physical network to host 1.1.1.2.
   5. The vSwitch on host 1.1.1.2 decapsulates it and delivers the original packet to VM-B.
6. VPC Isolation: If VM-A tries to access an IP address not belonging to VNI 1001, the vSwitch, when looking up the forwarding table, finds no matching rule or a rule indicating the traffic should be dropped, thus achieving VPC isolation.

9.1.4 Security Groups in VPC

Concept: A security group is a distributed, stateful virtual firewall that operates at the NIC level of a VM (or container). It allows users to define a set of inbound and outbound access control rules.

Implementation: Security group rules are also uniformly dispatched by the SDN controller and enforced on each host's vSwitch (or using Linux kernel's iptables/nftables/eBPF). Because it filters traffic at the first hop when entering or leaving the VM, it is very efficient and can achieve isolation between different VMs within the same subnet (while traditional network ACLs typically only control at the subnet boundary).

Through SDN and Overlay technology, we have successfully virtualized a shared physical network into multiple independent, secure, feature-rich VPCs. This is the network foundation for building a multi-tenant GPU cloud platform.

9.2 Cloud Load Balancing: The Pillar of Machine Learning Networks

In a GPU cluster, besides raw computing power for training, a very important application scenario is providing AI inference services. For example, deploying a trained image recognition model or language model as an online API service for external applications to call.

These inference services typically need to meet high availability and high scalability requirements. We cannot deploy just one service instance, as it might fail at any time and cannot handle high concurrency. Therefore, we typically deploy multiple identical service instances and use a load balancer to distribute external request traffic to these backend instances.

A cloud load balancer is a core network service component within a VPC.

9.2.1 Roles and Types of Load Balancers

Core Functions:

1. Traffic Distribution: Receives external requests and, based on certain scheduling algorithms (like round-robin, least connections, source IP hash), forwards requests to a healthy backend service instance.
2. Health Check: Periodically and proactively sends "heartbeat" probes (like an HTTP GET request or TCP SYN packet) to each backend service instance. If an instance does not respond correctly within a specified time, the load balancer marks it as "unhealthy" and temporarily stops forwarding new traffic to it.
3. High Availability: When a backend instance fails, the health check mechanism automatically detects this and removes it from the service pool, ensuring overall service availability.
4. Horizontal Scaling: When request volume increases, we only need to add more service instances to the backend pool. The load balancer automatically distributes traffic to them, achieving elastic scaling.
5. Single Entry Point: The load balancer provides a unified, fixed virtual IP address (VIP) as the service entry point for the entire backend service cluster, shielding clients from backend complexity.

Types by Working Layer:

1. Layer 4 Load Balancer (L4 LB):

   Operates at the transport layer of the OSI model (TCP/UDP).

   It makes forwarding decisions based on the four elements of the data packet: source IP, source port, destination IP, and destination port. It does not examine the application layer content (such as HTTP headers or URLs).

   Advantages: Very high performance, as it has simple processing logic and can be implemented with dedicated hardware or efficient kernel code.

   Disadvantages: Low flexibility; cannot perform fine-grained traffic scheduling based on application layer information.

2. Layer 7 Load Balancer (L7 LB):

   Operates at the application layer (e.g., HTTP/HTTPS).

   It can parse the content of application layer protocols. For example, an HTTP load balancer can decide which backend service cluster to forward a request to based on the request's URL path, domain name, cookies, HTTP headers, and other information. For instance, requests to api.example.com/images go to the image processing service, and requests to api.example.com/text go to the text processing service.

   Advantages: Extremely flexible; can implement complex routing and business logic. Can also provide advanced features like SSL offloading, content caching, and request rewriting.

   Disadvantages: Higher performance overhead, as it requires deep parsing and processing of each data packet.

9.2.2 Layer 4 Load Balancing Implementation: LVS/DR

In the Linux world, LVS (Linux Virtual Server) is a classic open-source project for implementing high-performance Layer 4 load balancing. Its most commonly used and highest-performing mode is Direct Routing (DR).

How LVS/DR Works:

1. Unified VIP: The load balancer (Director) and all backend real servers (RS) are configured with the same virtual IP address (VIP), but only the Director advertises this VIP to the external network.
2. Inbound Traffic (Request): A client sends a request packet to the VIP. The Director receives this packet, does not modify its IP header, but only modifies its Layer 2 MAC address to that of a selected backend RS. Then, the Director sends this "face-changed" packet directly into the same Layer 2 network where the RS resides.
3. Outbound Traffic (Response): The backend RS receives this packet, finds that its destination IP is the VIP configured on itself, and processes the request. After processing, the RS does not go through the Director; instead, it directly sends the response packet back to the client. Because the response packet's source IP is the VIP, the client can receive it normally.

Advantages of LVS/DR:

Extreme Performance: The Director only handles inbound request traffic and performs only a lightweight MAC address modification. Outbound response traffic, which is typically larger in volume, returns directly from the RS, completely bypassing the Director's resources. This allows LVS/DR to achieve very high throughput.

Implementation in Cloud Environments:

The Layer 4 load balancers of cloud providers (called NLB in AWS, CLB in Alibaba Cloud) mostly draw on the idea of LVS/DR at the bottom layer. However, through deep integration with SDN controllers and vSwitches, they implement more advanced features.

The vSwitch can play the role of the Director in LVS/DR, performing DNAT (modifying the destination IP to a selected RS's private IP) on inbound traffic, and then performing SNAT (modifying the source IP to the VIP) when the RS responds, achieving similar effects but with lower configuration requirements on backend RSs.

9.2.3 Layer 7 Load Balancing Implementation Technology

Layer 7 load balancing is essentially a reverse proxy.

Working Principle:

1. The L7 LB establishes a complete TCP connection and application layer session (such as HTTP) with the client.
2. It receives and fully parses the client's request.
3. Based on its configured routing rules, it then acts as a client to establish another brand new TCP connection and application layer session with the selected backend RS.
4. It sends the original (or modified) request to the RS.
5. The RS processes the request and returns the response to the L7 LB.
6. The L7 LB then returns the response to the original client through the first connection.

Common Implementations: Nginx, HAProxy, and Envoy are all excellent open-source Layer 7 load balancing and reverse proxy software. Cloud providers' Layer 7 load balancers (like AWS ALB, Alibaba Cloud ALB) are also typically built based on these open-source projects or similar self-developed technologies.

Integration in Kubernetes: Ingress

Kubernetes standardizes the usage of Layer 7 load balancing through the Ingress resource object and Ingress Controller.

Users create an Ingress object, defining routing rules based on hostnames and URL paths.

An Ingress Controller running in the cluster (e.g., nginx-ingress-controller) watches these Ingress objects and automatically translates these rules into its backend proxy software's (e.g., Nginx) configuration, dynamically applying these configurations.

9.2.4 Load Balancing Choices for AI Inference Services

For internal inter-service calls with extremely high performance requirements and simple protocols, Layer 4 load balancing can be considered.

For online inference services that need to provide external HTTP/HTTPS APIs, require flexible URL-based routing, or need advanced features like SSL offloading, Layer 7 load balancing is the more appropriate choice. In Kubernetes environments, using Ingress is the standard practice.

9.3 Dedicated Line Access, Peering Connections, and VPC Gateways

An isolated VPC has no value. We must connect the VPC to other external networks to build a complete hybrid cloud or multi-cloud architecture. VPC gateways are the key components for achieving these connections.

9.3.1 Scenario 1: Connecting to User's On-Premises Data Center (IDC)

Enterprises typically want to connect their cloud GPU cluster securely and stably with their own on-premises data center (IDC) to:

Migrate data between the IDC and the cloud VPC.

Allow cloud applications to access databases or services in the IDC.

Allow IDC employees to securely access the cloud development environment.

There are two main connection methods:

VPN Gateway + IPsec VPN:

Principle: Create a VPN gateway in the VPC and configure a VPN on the user's IDC edge router or firewall. One or more IPsec VPN tunnels are established between them over the public internet. All data transmitted in the tunnels is encrypted, ensuring security.

Advantages: Relatively simple and quick configuration, low cost (because it uses the public internet).

Disadvantages: Performance and stability depend on public internet quality; bandwidth is limited, latency is high and unstable, making it unsuitable for large-scale, continuous data transfer.

Direct Connect / ExpressRoute + Direct Connect Gateway:

Principle: The user rents a physical dedicated line through a telecom carrier, connecting one end to their IDC and the other end to the cloud provider's specified access point. Then, a Direct Connect gateway is created in the VPC to connect the VPC to the user's physical dedicated line through this access point.

Advantages: Provides a private, exclusive physical connection. High bandwidth (up to 10 Gbps or even 100 Gbps), extremely low and stable latency, and the highest security.

Disadvantages: High cost, long provisioning lead time.

Choice: For production environments requiring stable, high-performance hybrid cloud connectivity for enterprise-grade GPU clusters, dedicated line is the inevitable choice. VPN can serve as a temporary backup link for management purposes.

9.3.2 Scenario 2: Connecting to Other VPCs on the Same Cloud

In a large organization, different departments or projects may create multiple different VPCs. Sometimes, these VPCs need to communicate with each other.

VPC Peering Connection:

Principle: This is the most direct way to connect two VPCs. Users create a "peering connection" request between two VPCs. After both parties confirm, the cloud platform's SDN controller automatically establishes a private, high-bandwidth connection between the two VPCs' routers. The two VPCs can communicate directly using private IP addresses, as if they were in the same network.

Advantages: Good performance, low latency, as traffic is entirely transmitted within the cloud provider's backbone network.

Disadvantages: Peering connections are point-to-point and non-transitive. If VPC A peers with VPC B, and VPC B peers with VPC C, VPC A and VPC C cannot automatically communicate; they need a separate peering connection. When the number of VPCs is large, this creates a complex "mesh" topology that is difficult to manage.

Transit Gateway / Cloud Enterprise Network:

Principle: To solve the "mesh" problem of peering connections, cloud providers have introduced Transit Gateway (Cloud Enterprise Network) services. You can create a Transit Gateway and then "attach" all your VPCs and Direct Connect Gateways to this Transit Gateway.

The Transit Gateway acts like a cloud-based router or hub. Any network instance attached to it can communicate with any other attached instance, forming a hub-and-spoke topology.

Advantages: Greatly simplifies network management in multi-VPC, multi-region, hybrid cloud environments. It offers good scalability, and routing policies can be centrally configured.

9.3.3 Scenario 3: Connecting to the Public Internet

VMs or containers in a VPC need to access the internet or provide external services.

NAT Gateway:

Function: Provides outbound internet access for instances in private subnets without public IPs. When these instances access the public internet, their traffic is routed to the NAT gateway, which translates their source private IP to its own public IP (SNAT) before sending it out.

It is a highly available, high-throughput managed service, solving the performance and single-point-of-failure problems of self-built NAT hosts.

Internet Gateway (IGW):

Function: It is the "door" between the VPC and the internet. By attaching an IGW to a VPC and configuring a default route (0.0.0.0/0) pointing to the IGW, instances in subnets with public IPs can communicate bidirectionally with the internet.

Load balancers, NAT gateways, and other services that need to interact with the public internet all depend on the IGW's existence.

9.4 Implementation and Deployment of SDN NFV Gateways

All the VPC gateways (VPN gateways, NAT gateways, Direct Connect gateways, etc.) we discussed earlier, as well as load balancers and firewalls, are increasingly implemented in modern cloud environments through NFV (Network Function Virtualization) technology, as Virtual Network Functions (VNFs).

The idea of NFV is to implement traditional network functions (like routing, firewalls, and load balancing) that previously ran on dedicated hardware appliances in software, running them on standardized, general-purpose x86 servers. When NFV is combined with SDN, these VNFs become "software plugins" in the SDN network that can be centrally orchestrated and managed.

An SDN NFV gateway node is one or more high-performance servers dedicated to running these VNFs.

9.4.1 VM-Based NFV Deployment with virtio-net/vhost

This is a traditional approach to NFV, where VNFs are deployed in VMs.

virtio-net: A paravirtualized network device standard designed for KVM VMs.

Frontend Driver: Runs inside the Guest OS (VM), a lightweight network driver.

Backend Driver: Runs on the Host OS, typically implemented by QEMU.

Working Principle: The frontend driver places data packets into a shared memory ring buffer (virtqueue) with the backend, then notifies the backend to pick them up. Compared to emulating a real physical NIC (like e1000), virtio avoids a large number of VM-exit and VM-entry overheads, resulting in much better performance.

vhost-net: A further optimization. To avoid QEMU's user-space process becoming a bottleneck, vhost-net implements the virtio backend driver directly in the host's kernel. This allows the Guest OS to directly exchange data with the Host kernel, resulting in a shorter path and higher performance.

Advantages: Provides good isolation (VM level), mature and stable technology.

Disadvantages:

Performance Bottleneck: Even with vhost-net, data packets still need to go through multiple memory copies and context switches between the Guest kernel and Host kernel, making it difficult to meet 100 Gbps-level high-throughput requirements.

High CPU Overhead: Both Host and Guest CPUs are deeply involved in packet processing.

9.4.2 SR-IOV Based VM Deployment of NFV

To break through the performance bottleneck of virtio, SR-IOV technology can be applied to NFV gateway nodes.

Principle:

The gateway server uses a SmartNIC supporting SR-IOV.

The NIC is configured to create multiple Virtual Functions (VFs).

Each VF is directly passed through to a VM running a VNF.

Advantages:

Near-Bare-Metal Performance: The VNF VM can directly and exclusively access the NIC's hardware resources, completely bypassing the host's kernel and vSwitch, resulting in very low latency and very high throughput.

Disadvantages:

Low Flexibility: Loses many virtualization features. For example, VM live migration becomes difficult.

Limited Functionality: Advanced features provided by vSwitches (like security groups, traffic mirroring) cannot be directly applied to these directly passed-through VFs.

9.4.3 Accelerating NFV with DPDK

Whether based on virtio or SR-IOV, to squeeze the last drop of performance from the hardware, DPDK (Data Plane Development Kit) is needed.

DPDK in NFV: Inside the VM or container running the VNF, DPDK is used to take over the virtual NIC (virtio-net VF or SR-IOV VF) or physical NIC.

Core Technologies:

Kernel Bypass: DPDK applications directly read and write NIC queues in user space through polling mode, completely bypassing the kernel network stack's interrupts, system calls, and context switching overhead.

Huge Pages: Using 2 MB or 1 GB huge pages reduces TLB misses and improves memory access performance.

CPU Affinity: Binding DPDK's polling threads to specific CPU cores to avoid thread migration between cores.

Effect: Using DPDK, an ordinary x86 server core's network packet forwarding capability can increase by an order of magnitude, from millions of PPS (packets per second) to tens of millions of PPS, making it possible to achieve 100 Gbps line-rate VNFs with pure software.

Modern Implementation of SDN NFV Gateways:

Modern cloud network gateways are typically an integrated approach. They may run on bare-metal servers, use DPDK for extreme data plane performance, closely integrate with the SDN controller to receive forwarding tables and policies, and potentially use DPU/SmartNICs to hardware-offload the most CPU-intensive forwarding tasks (such as VXLAN encapsulation and decapsulation, connection tracking), achieving the best balance of performance, flexibility, and cost.

9.5 Chapter Summary

In this chapter, we completed the critical leap from physical networks to virtual networks. We recognized that network virtualization is an indispensable technical pillar for building a modern, secure, multi-tenant GPU cloud platform. It abstracts the underlying shared, rigid physical network into isolated, elastic, programmable virtual network services at the upper layer.

Our exploration began with the cornerstone of network virtualization -- SDN-based VPC technology. We understood SDN's core idea of "separation of control and forwarding" and how it achieves network resource pooling and logical isolation through Overlay networks (especially VXLAN), building an independent private network space for each tenant.

Next, we delved into the core network service components within a VPC:

Cloud load balancer, as the pillar of machine learning networks, especially AI inference services. We compared the ultimate performance of Layer 4 load balancing (like LVS/DR) with the application-layer flexibility of Layer 7 load balancing (like Nginx/Ingress), clarifying their selection criteria in different scenarios.

We learned how to connect the VPC internally and externally through VPC gateways. Whether connecting to the local IDC via VPN or dedicated line, connecting to other VPCs via peering connections or cloud enterprise networks, or connecting to the public internet via NAT gateways and internet gateways, these gateway components together form the bridge for the VPC to communicate with the world.

Finally, we delved into the low-level details, dissecting the engine that implements these advanced network functions -- the SDN NFV gateway. We saw the evolution of its implementation methods, from traditional VM-based deployment with virtio/vhost, to SR-IOV hardware pass-through for ultimate performance, to software acceleration using DPDK on general-purpose servers to squeeze out every bit of performance. We recognized that the modern NFV gateway is a complex system combining software and hardware, continuously pushing functionality downward and offloading it.

Through the study of this chapter, we are no longer just physical network operators but have become "planners" and "architects" of virtual networks. We have mastered a complete set of vocabulary and tools to describe, design, and implement a cloud-native network that meets the complex application needs of the AI era. This elastic, secure, programmable virtual network layer will closely integrate with the physical network, GPU virtualization, and other technologies we learned in previous chapters, jointly building a powerful, future-oriented large model computing infrastructure.