祝融说。

After eleven chapters of painstaking effort, we have successfully designed and built a functionally complete, technologically advanced computing infrastructure for large models. We now possess a computing cluster of top-tier GPU servers, a communication artery formed by a high-speed lossless network, a formidable data foundation built from layered heterogeneous storage, and a cloud-native, microservices-based application development and runtime platform. We have erected this magnificent "AI metropolis."

Yet the completion of the city is only a beginning. How to manage and operate this city scientifically, efficiently, and with precision, keeping it running safely, stably, and economically 24/7 while continuously creating value for its "citizens" (users), is a far longer and more arduous challenge. It calls for a powerful city management system.

In this chapter, we enter the highest tier of AI infrastructure---management and operations. From the vantage point of a cloud platform product manager and a senior operations expert, we will explore how to build a comprehensive, automated management and operations platform for a large-scale GPU cluster. This platform is no longer a single technical component; it is an integrated system of governance made up of three core sub-platforms---operations, business operations, and auditing. It will help us shed the role of "builder" and ultimately become a "mayor" who commands the whole picture.

We will examine the design and implementation of these three platforms in turn:

1. Cloud Operations Platform: This is the "municipal engineering and emergency center" that keeps the cluster running stably. We will explore how to automate the management of vast hardware infrastructure, how to build a panoramic monitoring and alerting system that spans every layer, and how to establish an authoritative configuration management database (CMDB) that serves as the bedrock of all operations activities.
2. Cloud Business Operations Platform: This is the "commerce and finance center" that converts resources into value. We will learn how to meter and bill precisely for precious resources such as GPUs, how to achieve fair, efficient resource allocation through multi-tenancy and resource quotas, and how to provide a self-service portal that improves user experience and operational efficiency.
3. Cloud Auditing Platform: This is the "inspection and security center" that ensures the cluster's compliance and safety. We will explore how to record and audit every action taken by users and administrators, meeting security and compliance requirements while enabling traceability when problems arise.

By the end of this chapter, you will master a complete, enterprise-grade methodology for cloud platform governance. You will understand that a successful AI cloud platform is not a mere pile of technology but an organic synthesis of process, standards, automation, and precise operations. This is the final mile of your journey from a purely technical expert to a senior architect or technical leader capable of overseeing massive infrastructure end to end, balancing technology and business, and closing the loop of value creation.

12.1 Cloud Operations Platform

The cloud operations platform is the cornerstone of the entire cluster management system. Its core mission is to safeguard the stability, availability, and health of the infrastructure while raising the efficiency and automation of operations work. In a large-scale GPU cluster of thousands of servers, tens of thousands of NICs, and hundreds of thousands of cables, the traditional mode of "logging into a jump host and running commands" is unimaginable. We must build a powerful, automated operations platform, composed chiefly of the three core components below.

12.1.1 Hardware Infrastructure Management

This is the most foundational capability of the operations platform, responsible for automating management over the entire lifecycle of physical hardware.

Challenges:

• Massive scale: How do we simultaneously manage the racking, installation, configuration, and repair of hundreds or thousands of servers?
• Heterogeneity: The cluster contains GPU servers, storage servers, management servers, and many other hardware models of different types.
• Complex state: Every device carries firmware (BIOS, BMC, NIC firmware, etc.) that must be managed and upgraded.

Core functions and implementation technologies:

Asset Discovery and Registration

• Automated discovery: When a new server is connected to the out-of-band management network, the platform should automatically discover the device through protocols such as LLDP and DHCP, and obtain its basic information, including MAC address and serial number.
• Integration with CMDB: The discovered device information should be entered into the CMDB automatically or semi-automatically, completing asset initialization.

Automated OS Provisioning

PXE (Preboot Execution Environment): This is the core technology for automated installation onto bare-metal servers.

Workflow:

1. An administrator assigns an installation template to a new server on the platform (for example, "DGX OS for H100").
2. The platform configures the server's BMC to boot from the network.
3. When the server boots, its NIC issues a PXE boot request.
4. The DHCP server on the network responds, telling it the TFTP server address and the filename of a boot loader (such as iPXE).
5. The server downloads and executes the boot loader from the TFTP server.
6. Following the instructions it obtains from the platform, the boot loader downloads the operating system kernel, the initrd image, and automated installation configuration files (Kickstart/Preseed) from an HTTP or NFS server, then launches an unattended, automated installation.

Through this pipeline, we can install operating systems in parallel and automatically for a rack of servers within minutes.

Configuration Management Automation

Tools: Ansible, SaltStack, Puppet, and Chef are the mainstream configuration management tools of the industry. Among them, Ansible is especially popular in automated operations because it is agentless, operates over SSH, and uses a simple YAML language.

Working mode (using Ansible as the example):

1. An administrator writes a Playbook. A Playbook is a YAML file that declaratively describes the "final state" a server should be in (for example, which packages should be installed, what the configuration files should contain, and which services should be started).
2. The Ansible engine reads the Playbook, connects to the target server (or server group) over SSH, and performs a series of operations to bring the server's state into line with what the Playbook describes.
3. Ansible is idempotent: a Playbook can be rerun any number of times, but change operations are actually executed only when the server's current state differs from the target state.

Applications: We can use Ansible to automate all configuration tasks---system initialization (setting hostname, network, and NTP), GPU driver installation, deployment of monitoring agents, distribution of SSH keys, and so on.

Firmware Management and Out-of-Band Operations

• Through the out-of-band management network, the platform should be able to call the Redfish or IPMI APIs of servers' BMCs to perform remote power-on/off, restart, hardware log inspection, and virtual media mounting.
• The platform should also support batch firmware query and upgrade---for example, scanning the BIOS or BMC firmware versions of all servers in the cluster and automatically upgrading, in batches, those that are outdated.

Open-source and commercial solutions:

• Open-source stack: One can build a self-developed hardware automation management platform on the basis of open-source tools such as iPXE + Kickstart/Ansible + Cobbler/MAAS.
• Commercial solutions: Red Hat Satellite, SUSE Manager, and others provide mature solutions.

12.1.2 System Monitoring and Alerting Platform

If hardware management is "installation," then monitoring and alerting is "health examination" and "first aid." A panoramic monitoring platform is the "eyes" and "ears" that keep the cluster stable and let us spot and localize problems quickly.

The core principle of monitoring design: layering and correlation

A good monitoring system must cover every layer, from the underlying physical hardware to the AI applications above, and must be able to correlate data across these layers into a complete fault-diagnosis chain.

• Physical layer: Server hardware (temperature, power, fans), switches (port status, traffic), and PDUs.
• System layer: Operating system metrics (CPU utilization, memory, disk I/O, network traffic).
• GPU layer: Core metrics of GPU cards (GPU utilization, memory utilization, temperature, power, NVLink traffic, MIG instance status).
• Platform layer: Kubernetes cluster state (node status, Pod count, API Server latency) and middleware state (Kafka lag, Redis hit rate, database connection count).
• Application layer: AI task metrics (training loss, throughput in samples/sec, inference latency P99).

The de facto standard for cloud-native monitoring: Prometheus + Grafana

Prometheus: A powerful, open-source time series database (TSDB) and monitoring system.

• Pull model: Prometheus periodically and proactively pulls metric data from HTTP endpoints (usually /metrics) exposed by the monitored targets.
• Exporter: To allow systems that do not natively speak Prometheus's format to be monitored, the community has developed a great many Exporters. An Exporter is a small conversion program that collects data from a target system, converts it into a format Prometheus can understand, and exposes it through an HTTP endpoint. For example:
  • node-exporter: collects the core metrics of an operating system.
  • dcgm-exporter: NVIDIA's official Exporter for collecting detailed GPU metrics, built on the DCGM (Data Center GPU Manager) library.
  • kube-state-metrics: collects the state of the various objects in a Kubernetes cluster.
• The powerful query language PromQL: Prometheus provides an extremely capable query language, PromQL, for flexibly querying, aggregating, and computing over the collected time-series data.
• Alertmanager: Prometheus computes alert states from predefined alerting rules and then sends alert events to Alertmanager. Alertmanager is responsible for deduplicating, grouping, and silencing alerts, and for delivering notifications through email, Slack, DingTalk, phone, and other channels.

Grafana: An open-source visualization platform of enormous richness.

• Data sources: Grafana can connect to many data sources, the most important of which is Prometheus.
• Dashboards: In Grafana, users can create all manner of compelling dashboards with simple clicks and drags. Each dashboard is composed of multiple panels; every panel runs a PromQL query and renders the result as a chart, gauge, table, or other form.

We can build a dedicated monitoring dashboard for each layer of the GPU cluster (hardware, system, GPU, application), bringing global state into full view.

The complement of logging and tracing:

• Centralized logging (EFK/Loki): As discussed in Chapter 11, this provides full-text search over application and system logs---another powerful ally in troubleshooting.
• Distributed tracing (Jaeger/Zipkin): For microservice architectures, distributed tracing can track the complete call chain of a request across many services, helping us quickly locate performance bottlenecks and the exact stage where errors occur.

By integrating the "three pillars of observability"---Metrics (Prometheus), Logging (Loki/EFK), and Tracing (Jaeger)---we can build a powerful, three-dimensional monitoring and alerting system in which not the slightest anomaly can hide.

12.1.3 Configuration Management Database (CMDB)

The CMDB is the "central nervous system" and "authoritative data source" of the entire operations platform. It is not merely an Excel spreadsheet that records asset information; it is a living, dynamic database that describes and manages every component (Configuration Item, CI) of the IT infrastructure and the relationships among them.

What must the CMDB manage?

• Hardware CIs: data centers, computer rooms, racks, servers (model, serial number, CPU, memory, GPU cards), switches, PDUs, and so on.
• Software CIs: operating systems (version), IP addresses, hostnames, deployed applications, database instances, middleware, and so on.
• Relationships: The relationships among CIs are the very essence of a CMDB. For example:
  • GPU server gpu-server-01 is located in rack Rack-A01.
  • gpu-server-01 has DGX OS 5.0 installed.
  • gpu-server-01 is connected to port eth0 of switch leaf-01.
  • Application recommend-service is deployed on gpu-server-01.

The role of the CMDB:

1. Single source of truth for operations: Automation scripts (such as Ansible Playbooks) can dynamically fetch from the CMDB the list of target servers and their variables. Monitoring systems can pull device metadata from the CMDB, enriching alerts with far more context---so that an alert identifies not merely IP 10.1.1.1 but "GPU server 01 of the recommendation system in rack A01."
2. Change management and impact analysis: When a change is needed to a component (such as upgrading switch firmware), querying the CMDB lets us quickly analyze which servers and applications the change might affect, so that a thorough change plan can be drawn up.
3. Fault localization: When an application alerts, we can trace quickly along the CMDB's relationship chain back to the servers, network, and storage it depends on, narrowing the troubleshooting scope.
4. Cost and capacity management: The CMDB is the underlying data source for cost accounting and capacity planning.

Building the CMDB:

Building a successful CMDB is a complex undertaking. The key lies in guaranteeing the accuracy and timeliness of its data.

Automated discovery is the critical step: CMDB data should be discovered and synchronized automatically, as far as possible, from the various management systems (such as the hardware management platform, Kubernetes, and vCenter) through automation scripts, rather than depending on manual entry.

• Open-source options: Ralph, Collins, iTop, and so on.
• Commercial options: ServiceNow, BMC, and others provide powerful ITSM/CMDB solutions.

A cloud operations platform that integrates automated hardware management, panoramic monitoring and alerting, and an authoritative CMDB will let us manage an ultra-large-scale GPU cluster with composure and efficiency, making the transition from a "firefighting crew" to a "preventive maintenance specialist."

12.2 Cloud Business Operations Platform

If the operations platform is concerned with "machines" and "stability," the business operations platform is concerned with "people" and "value." Its core goal is to package the underlying, raw computing, storage, and network resources into "cloud services" that users can readily understand and use, then to meter, bill, and manage the consumption of those services, ultimately achieving fair resource allocation and turning value into revenue.

12.2.1 Multi-Tenancy and Resource Quota Management

In a shared GPU cluster, different users, teams, or projects must be isolated and resource-limited, so as to prevent abuse of resources and unfair preemption.

• Tenant: A basic isolation unit in the business operations platform, which may correspond to a department, a project team, or an external customer.
• Resource quota:
  • The platform must set an upper cap on the total amount of each resource a tenant may use. This is like allocating each household a monthly allowance of water and electricity.
  • Implementation in Kubernetes: Kubernetes provides the ResourceQuota and LimitRange objects natively.
  • ResourceQuota can limit the total resources within a namespace (a tenant typically maps to one or more namespaces), for example:
    • requests.cpu: "100" (the total CPU requests of all Pods in this namespace may not exceed 100 cores)
    • limits.nvidia.com/gpu: "32" (the GPU usage cap is 32 cards)
    • count/pods: "1000" (the Pod count cap)
  • LimitRange can set default, minimum, and maximum resource requests and limits for each Pod or container in a namespace.
• Through fine-grained quota management, resources can be allocated equitably among tenants, preventing a handful of "deep-pocketed" users from exhausting the entire cluster's resources.

12.2.2 Metering and Billing System

Metering and billing is the heart of a cloud platform's business loop. It must measure precisely how much of each resource every tenant consumes, and, per the agreed pricing strategy, compute the corresponding charges.

The challenges of metering:

• Diverse resource types: What must be metered goes beyond GPU card-hours to include CPU core-hours, memory GB-hours, storage GB-months, and network traffic in GB.
• The complexity of GPU metering:
  • For exclusively dedicated GPUs, we can meter by "card x hour."
  • For GPUs shared through technologies such as MIG or cGPU, how do we fairly meter the use of compute power and memory? By the allocated share, or by actual usage? This must be tightly integrated with the underlying virtualization scheme.
  • For example, the management daemon of a cGPU solution must periodically report each container's actual compute and memory consumption.

Collection and processing of metering data:

1. Data sources:
   • Pull cumulative usage of CPU, memory, network, and other metrics from Prometheus.
   • Obtain GPU usage data from GPU Exporters (such as dcgm-exporter) or from virtualization management daemons.
   • Obtain storage capacity usage from the management interfaces of storage systems.
2. Data processing pipeline:
   • A backend data-processing system (for example, one built on Spark or Flink) periodically (say, hourly) pulls this raw metering data.
   • The data is cleansed, aggregated (for example, minute-level samples are aggregated into hourly usage), and correlated (resource usage is associated with tenant IDs).
   • The processed, normalized billing data is stored in a dedicated database.

Billing and invoice generation:

• Pricing strategy: The business operations platform must allow administrators to define a unit price for each resource (such as gpu-a100-hour, cpu-core-hour). Different tenants can also be granted different discounts.
• Invoice generation: A billing engine periodically (say, daily or monthly) scans the billing database, computes each tenant's charges according to the pricing strategy, and generates detailed invoices.

12.2.3 User Self-Service Portal

To improve user experience and operational efficiency, the platform must provide end users with a web-based self-service portal. Users no longer need to submit tickets or contact administrators to request resources.

Core functions:

• Dashboard: After logging in, users can view their tenant's quota usage, current charges, historical invoices, and so on.
• Service catalog and requests:
  • The platform presents standardized services (such as "a JupyterLab development environment with two A100 GPUs and 1 TB of storage" or "a ten-node PyTorch distributed training cluster") as "products" arrayed in a service catalog.
  • Like shopping on an e-commerce site, users can select the service they need, fill in parameters (such as image version and code repository address), and "place an order with one click."
• Lifecycle management: Users can view, start, stop, and destroy the service instances they have requested.
• Approval workflow: For costly resource requests, an approval workflow can be integrated. Only after a user's request has been approved by their supervisor or budget owner will the platform actually provision the resources.

Backend implementation:

• The backend of this portal is typically a microservice application that integrates with all backend systems---Kubernetes, the CMDB, the metering and billing system, and so on---through their APIs.
• When a user places an order, the portal translates the request into a series of API calls to the backend systems (for example, creating a Namespace, a Deployment, and a Service in K8s).

A friendly, automated service portal is essential to raising user satisfaction and stickiness, freeing enormous amounts of operations manpower, and achieving operations at scale.

12.3 Cloud Auditing Platform

As the cluster grows and the number of users increases---especially when the platform must satisfy the compliance requirements of industries such as finance and healthcare---recording and auditing every operational action becomes critical.

The goals of auditing:

1. Security compliance: Satisfy the traceability requirements of regulations such as SOX, HIPAA, and GDPR.
2. Post-event tracing: When a security incident or a major failure occurs, the audit logs let us quickly trace back precisely who (Who) did what (What), when (When), from where (Where), and with what result (Result).
3. Risk discovery: Analysis of audit logs can surface anomalous behavior patterns (for example, a user repeatedly attempting to access unauthorized resources late at night) and trigger alerts.

What must be audited:

• Access to the platform: every user login to the portal and every API call to the platform.
• Calls to the Kubernetes API: every create, delete, or modify operation on K8s resources (Pod, Service, Secret, and so on) via kubectl or the API. The Kubernetes API Server supports powerful audit logging natively.
• Access to servers: every command executed after logging into a server over SSH.
• Key operations on cloud services: modifying firewall policies, deleting an important storage volume, and so on.

The architecture of the auditing platform:

1. Log collection: Configure every key component (the platform API gateway, the Kubernetes API Server, the bastion host, the cloud service controller) to send its audit logs, in structured form, to a centralized logging system.
2. Centralized storage: These highly sensitive audit logs should be stored in a secure, tamper-proof system with a long-term retention policy. A dedicated security information and event management (SIEM) system can be used, or an independent Elasticsearch/Loki cluster with strict access controls.
3. Analysis and alerting: A real-time analysis engine continuously scans the audit log stream. Based on predefined audit rules (for example, "any read of a production Secret should trigger an alert immediately"), it detects suspicious behavior and raises alerts.
4. Query and reporting: Provide a dedicated query interface so that security and audit personnel can conveniently search and analyze historical audit logs and generate compliance reports.

A complete cloud auditing platform is the last line of defense for the security and compliance of the entire AI cloud platform, and the foundation for building trust with users and regulators alike.

12.4 Chapter Summary

In this chapter, we completed the "final piece of the puzzle" in building an enterprise-grade AI infrastructure---the management and operations platform. We recognized that the value of a successful cloud platform lies not only in the sophistication of its underlying technology but in the completeness of its governance layer above. We decomposed this governance system into three sub-platforms that complement one another and that none can do without.

1. We built the Cloud Operations Platform, the "municipal engineering center" that safeguards the cluster's stability and health.

   • Through automated management of the hardware infrastructure (based on PXE and Ansible), we achieved unattended delivery from bare metal to a usable operating system.
   • Through a panoramic monitoring and alerting system centered on Prometheus and Grafana, we fitted the cluster with "eyes" and "nerves" spanning every layer, achieving end-to-end observability from hardware to application.
   • Through an authoritative, automated CMDB, we provided a "single source of truth" for all operations activities, systematizing configuration, change, and fault management.

2. We built the Cloud Business Operations Platform, the "commerce and finance center" that closes the loop of resource value.

   • Through multi-tenancy and resource quota management, we achieved fair, isolated resource allocation.
   • Through a fine-grained metering and billing system, we learned to measure and quantify the value of resource usage.
   • Through a self-service portal, we greatly improved user experience and operational efficiency, completing the shift from a "handicraft workshop" to a "cloud service supermarket."

3. We built the Cloud Auditing Platform, the "inspection center" that ensures the platform's security and compliance. By recording and analyzing every critical action, we provided traceable assurance for the platform's secure operation and met enterprise-level compliance requirements.

At this point, we have grown from hardware engineers, software engineers, network engineers, and storage engineers into chief architects and platform operations officers capable of commanding a complex, large-scale AI cloud platform in its entirety. We know not only how to "build" this AI metropolis but also how to "govern" it---how to keep it running stably, how to make it create value, and how to keep it secure and compliant.

In the book's final chapter, we will bring all the theoretical knowledge of the preceding chapters into a cohesive practice through a concrete implementation case, fully demonstrating how to design and implement an end-to-end GPU computing platform from scratch for a real machine learning application (such as training an autonomous driving model). This will be the ultimate test and the crowning achievement of our entire learning journey.