祝融说。

In the previous three chapters, we have respectively explored the planning of infrastructure and the technical support for training and inference platforms. This is like having meticulously designed and built a fully functional, high-performance "Computing Skyscraper." However, after the building is completed, an even more arduous and enduring challenge begins — how to ensure this building's lights are on 24/7, its elevators run smoothly, its pipelines are trouble-free, and it can withstand wind, rain, thunder, and even earthquakes? This is the core mission of platform operations and high-availability assurance.

Operations is the mark of the computing platform's transformation from a "construction project" to an "online service," the fundamental guarantee for the platform's continuous value delivery. A platform lacking strong operational capabilities, no matter how advanced its technical architecture, can be paralyzed by a single minor fault, causing incalculable losses to the business. Especially in a critical infrastructure field like State Grid, the stability of the computing platform is directly related to the safe and stable operation of the power grid and the continuity of service — its importance cannot be overstated.

This chapter focuses on the bottom-line assurance link within the technology dimension of the "Five-in-One" construction approach, fully deconstructing our platform operations and high-availability assurance system. This system aims to achieve one core goal: transforming from a passive "fire brigade" style of fault handling to an active "health care doctor" style of health management and a "special forces" style of resilience building. We will explore three progressively deepening themes:

1. "Rapid Response" — When problems occur: How to establish an efficient problem localization and handling process, capable of quickly finding the root cause amidst complex hardware and software faults and restoring service? (8.1 Computing Operation Problem Localization and Handling)
2. "Prevention Before the Fact" — Before problems occur: How to build a scientific resource health assessment system capable of proactively perceiving and warning about the platform's "sub-healthy" state, nipping a large number of problems in the bud? (8.2 Computing Resource Health Assessment System)
3. "Forged Through Repeated Practice" — Preparing for extreme situations: How to proactively build system-level high reliability through redundant design, stress testing, and emergency drills, ensuring the platform still possesses strong survival and recovery capabilities when facing major disasters? (8.3 High-Availability Assurance)

These three links together form a deep operations assurance matrix from "tactical response" to "strategic defense," the "ballast stone" for the entire computing platform to proceed steadily and safely into the future.

Computing Operation Problem Localization and Handling (Hardware, Software, Configuration)

Faults are the norm in operations. A large-scale computing platform with thousands of servers, tens of thousands of GPU cards, and a complex software stack can have faults originating from a myriad of sources — physical damage to hardware, logical defects in software, or subtle configuration errors. The ability to localize and handle problems directly determines the platform's Mean Time To Recovery (MTTR), which is the most core, fundamental "combat capability" of the operations team.

Build a Layered, Domain-Specific Fault Localization Methodology

Faced with a complex fault phenomenon (for example, "my training task has become slower"), the most taboo approach is a headless fly-style, aimless troubleshooting. We must establish a structured, top-down fault localization methodology to quickly narrow down the scope of the problem.

Top-Down, Layer-by-Layer Investigation

Layer 1: Application and Model Layer

• Preliminary judgment: Is the problem an isolated case or common? Is only one user's task slower, or are all users' tasks slower? If the former, the problem is likely at the application layer.
• Investigation checklist:
  • Code changes: Has the user recently modified the code? (The most common reason)
  • Data issues: Is there something abnormal with the input data? Is the data loading path correct?
  • Dependency environment: Has the version of the dependent Python library changed?
  • Algorithm issues: Has the model entered an abnormal state such as non-convergence or gradient explosion?

Layer 2: Platform and Scheduling Layer (Kubernetes)

• Preliminary judgment: If multiple applications have problems simultaneously, or new tasks cannot be scheduled, the problem may be at the platform layer.
• Investigation checklist:
  • Pod status: Use kubectl describe pod to view the Pod's events. Are there errors such as image pull failure, health check failure, or resource exhaustion (OOMKilled)?
  • Scheduler status: Check the scheduler logs. Are there tasks in a prolonged Pending state due to insufficient resources or unmet scheduling policies?
  • Core component health: Check whether the core components of the Kubernetes Master nodes (kube-apiserver, kube-scheduler, etcd, etc.) are healthy.

Layer 3: System and Software Layer (OS & Drivers)

• Preliminary judgment: If all Pods on a particular physical node are failing, or the node is marked NotReady by the scheduler, the problem may be at the system layer.
• Investigation checklist:
  • System logs: Use dmesg or journalctl to check for kernel errors, I/O errors, etc.
  • GPU driver status: Use nvidia-smi to check whether all GPU cards are present and their status is normal. Check the status of services such as NVIDIA Fabric Manager.
  • Network configuration: Check whether the node's IP address, routing, and DNS configuration are correct.

Layer 4: Hardware and Infrastructure Layer

• Preliminary judgment: If clear hardware errors (such as MCE - Machine Check Exception) are seen in the system logs, or nvidia-smi shows a GPU card as "missing," the problem is clearly at the hardware layer.
• Investigation checklist:
  • Out-of-band management logs (BMC/iDRAC): This is the "golden key" for diagnosing hardware problems. Log into the server's out-of-band management interface to view the hardware event log, which can usually precisely pinpoint which memory module, fan, or power module is faulty.
  • Switch logs: Check if the switch port connected to the problem node has a large number of error packets or packet loss records.
  • Physical inspection: If remote diagnosis fails, on-site personnel in the machine room are needed for a physical inspection, such as checking indicator light status, reseating cables or cards.

Build Standardized Fault Handling Processes (SOP)

We solidify the above methodology into standardized operating procedures (SOP) for different fault types.

• Fault classification: Classify faults into P1 (Emergency), P2 (Important), and P3 (General) levels based on impact scope and business importance, with different levels corresponding to different response SLAs and escalation paths.
• SOP content: Each SOP includes:
  • Description of the fault phenomenon.
  • Common possible causes.
  • Step-by-step investigation guide (Checklist).
  • Standard recovery operations.
  • Escalation path: If the SOP cannot resolve the problem, who should it be escalated to (e.g., second-line expert, vendor).
• Case: "GPU Card Drop Fault Handling SOP"
  1. Phenomenon: nvidia-smi cannot see the GPU, or displays ERR!.
  2. Investigation:
     • dmesg | grep NVRM to check GPU driver error logs.
     • Log into BMC to check hardware logs, confirming whether there is a PCIe link error.
     • On the same machine, swap the problematic GPU with a normal GPU in the slot to determine whether it's a card problem or a motherboard slot problem.
  3. Recovery:
     • Try a cold boot of the server.
     • If confirmed as a physical GPU card fault, immediately create a hardware repair ticket and obtain a replacement card from the spare parts inventory for replacement.
     • After replacement, perform stress testing on the new card to confirm stability before putting it online.

The "Three Axes" for Improving Problem Handling Efficiency

Methodology and processes alone are not enough; we also need tools and mechanisms to empower, maximizing handling efficiency.

First Axe: Unified Observability Platform

• End of data silos: Aggregate all monitoring data mentioned earlier — Metrics, Logs, Traces — into a unified, interactive platform (such as one built on ELK or Grafana Loki/Tempo/Mimir).
• Context correlation: Operations personnel can simultaneously view performance metric curves around the time of a fault, related application logs and system logs, and even distributed call traces on a single interface. This "context integration" capability greatly shortens the time spent switching between different systems and manually correlating information. It is the "killer feature" for achieving rapid localization.

Second Axe: AIOps Intelligent Operations

• Anomaly detection: Use machine learning algorithms to model massive monitoring indicators, automatically detecting those subtle abnormal fluctuations or trend changes that are difficult to detect with the human eye, upgrading from "threshold alerting" to "intelligent warning."
• Root cause analysis: When multiple alerts occur concurrently, the AIOps system can, based on historical data and topological relationships, automatically infer the most likely "root cause alert," helping operations personnel cut through the storm of alerts and get to the heart of the problem.
• Fault self-healing: For some patterned, deterministic faults, authorize the AIOps platform to automatically execute recovery operations (such as restarting Pods, isolating nodes), achieving unattended operations.

Third Axe: Efficient Collaborative Combat Mechanism

• ChatOps: We push operational events such as alerts and tickets in real time to dedicated operations collaborative groups (such as WeCom, DingTalk) through bots. Operations personnel, developers, and vendor support personnel can communicate directly, execute commands, and share logs within the group, with all discussions and operations automatically saved, forming a transparent and efficient "war room."
• War Room mechanism: For P1-level major faults, we immediately activate the "War Room" mechanism, requiring all relevant parties (including senior managers) to join an online conference within 15 minutes, concentrating all resources until the problem is resolved.

Computing Resource Health Assessment System

Problem handling is "repairing the sheep pen after the sheep have been lost," while health assessment is "keeping the sheep pen door closed." Its core idea is to shift focus from "point"-like faults to "surface"-level health trends. Through a set of quantitative, multi-dimensional assessment models, continuously and proactively evaluate the health status of the platform and resources, and based on the assessment results, carry out preventive maintenance and optimization.

Core Concepts and Models of Health Assessment

• From "Available" to "Healthy": A resource being "available" (pingable, logable) does not mean it is "healthy" (it may have severely degraded performance or potential risks). Health is a more comprehensive, more forward-looking concept.
• Quantify and score: We quantify the vague concept of "health" into a calculable "health score" from 0-100. Below 60 is "unhealthy," 60-80 is "sub-healthy," and above 80 is "healthy."
• Multi-dimensional weighted model: The health score is obtained by weighting the average of sub-indicators from multiple dimensions based on their importance. Our model mainly includes four dimensions: stability, performance, resource efficiency, and security.

Detailed Health Assessment Indicator System

We establish an independent health assessment model for each type of core resource in the platform (computing nodes, GPU cards, storage clusters, network equipment).

Computing Node Health Model

• Stability (weight 40%):
  • Hardware error rate: Frequency of serious hardware errors recorded in BMC logs, such as memory ECC errors, CPU MCE errors, PCIe link errors.
  • Kernel crash count: Number of Kernel Panics per unit time.
  • Unplanned restart count: Number of unplanned restarts.
  • GPU card drop count.
• Performance (weight 30%):
  • Benchmark performance score: Regularly run standard benchmark tests for CPU (SPEC CPU), memory (Stream), and disk (Fio), comparing test results with the baseline for the same model of equipment to obtain a performance score. Performance degradation is an important sign of hardware aging.
  • Stability under high load: Whether there are phenomena such as overheating throttling or performance jitter during stress testing.
• Resource efficiency (weight 20%):
  • Average resource utilization: Long-term average utilization of CPU and memory. Too low utilization may indicate resource waste or uneven scheduling.
  • Degree of resource fragmentation.
• Security (weight 10%):
  • Security baseline compliance: Whether the node's OS configuration, kernel parameters, and open ports comply with the company-defined security baseline standards.
  • Number of high-risk vulnerabilities: Number of unpatched high-risk security vulnerabilities found through scanning.

GPU Card Health Model

• Stability (weight 50%):
  • ECC error count: The most critical indicator for measuring GPU memory health. We differentiate between correctable (Single-Bit) and uncorrectable (Double-Bit) errors. The appearance of the latter typically means the memory chip is about to fail.
  • Temperature and power consumption: Whether the GPU's long-term operating temperature and power consumption are within normal ranges. Sustained high temperature is the main cause of reduced GPU lifespan.
  • Xid errors: The number of various internal errors recorded by the NVIDIA driver.
• Performance (weight 40%):
  • Computing performance score: Regularly run benchmark tests such as matrix operations in CUDA Samples to evaluate the performance of its computing units.
  • P2P/NVLink bandwidth: Test whether the communication bandwidth with other GPU cards on the same node meets standards.
• Resource efficiency (weight 10%):
  • Logical utilization: The average true utilization rate of its computing units when allocated and in use.

Implementation and Application of Health Assessment

• Automated assessment platform: We developed an automated health assessment platform.
  • Data collection: The platform regularly (e.g., daily) automatically collects all raw indicators needed for calculating the health score from multiple data sources such as the monitoring system, CMDB, and security scanning system.
  • Score calculation and display: The platform automatically calculates the health score for each resource based on the preset weight model, and displays it through visual dashboards (such as heat maps, trend charts). Operations personnel can see the "health map" of the entire cluster at a glance.
• Operations decisions based on health:
  • Preventive maintenance: The platform automatically generates a "Low Health Score Resource List." For resources with consistently low scores in a "sub-healthy" state, the operations team proactively intervenes, conducting in-depth diagnosis and preventive maintenance (such as replacing fans, reapplying thermal paste, upgrading firmware) to prevent them from evolving into real faults.
  • Automated isolation: When a resource's health score falls below a certain danger threshold (e.g., a GPU showing numerous uncorrectable ECC errors), the system can automatically isolate it from the production environment (marked as unschedulable) and create a repair ticket.
  • Optimized scheduling strategy: The scheduler can integrate resource health score information, prioritizing selection of nodes with higher health scores when scheduling tasks, and avoiding scheduling key tasks to "sub-healthy" nodes.
  • Guiding hardware procurement and retirement: Through statistical analysis of long-term health data of hardware from different brands and batches, provide data-driven decision-making basis for future hardware procurement selection and old equipment retirement plans.

By establishing this scientific, quantitative health assessment system, our operations work has achieved a fundamental transformation from "passive response" to "active management," significantly improving the overall stability and service quality of the platform.

High-Availability Assurance: Redundancy, Stress Testing, and Emergency Drills

Health assessment can address routine, single-point "minor ailments." But to withstand destructive, regional "major disasters" (such as a complete power outage in an entire machine room, failure of a core switch), powerful high-availability and disaster recovery capabilities must be built at the system architecture level. This is an active, systematic engineering effort aimed at improving the platform's "resilience."

Architecture-Level Redundant Design: No Single Point of Failure

The core principle of high-availability design is "redundancy" — ensuring through backups and backups of backups that there is no single point of failure in the system that would cause the entire system to collapse.

• Infrastructure redundancy:
  • Power redundancy: Use dual utility power feeds, equipped with N+1 or 2N configured UPS (Uninterruptible Power Supply) and diesel generators.
  • Network redundancy:
    • Equipment redundancy: Core switches and aggregation switches adopt dual-machine hot standby (e.g., VPC/MLAG).
    • Link redundancy: Servers use dual NICs, dual uplinks, connected to two different access switches respectively.
• Platform management node redundancy:
  • Kubernetes Master node high availability: Adopt a "three-master" or "five-master" deployment mode, with the etcd database performing data synchronization and elections across multiple master nodes. The failure of any single master node does not affect the management functionality of the entire cluster.
  • Critical add-on component high availability: Key platform components such as Prometheus and CoreDNS adopt multi-replica deployment modes.
• Application service redundancy:
  • Cross-node/cross-rack deployment: Through Kubernetes Pod anti-affinity policies, forcing multiple replicas of a service to be distributed deployed across different physical servers, different racks, or even different availability zones, preventing a single server or rack failure from causing the entire service to be unavailable.

Stress Testing: Finding the Platform's Limits and Inflection Points

Redundant design provides theoretical high availability, but whether the system can run stably under real high loads, and what its performance limit is, must be verified through real-world stress testing.

• Full-chain stress testing: We regularly organize full-chain stress testing activities.
  • Simulate real traffic: Use tools such as Locust and JMeter to simulate thousands of virtual users, initiating sustained, incremental concurrent requests to core training and inference services from the application entry point (such as the API gateway).
  • Find performance inflection points: We continuously increase pressure until some part of the system (perhaps the database, a microservice, or GPU resources) reaches a bottleneck, leading to a sharp increase in response time or a spike in error rates. This point is our system's "performance inflection point."
• Capacity planning and performance optimization:
  • Through stress testing, we can precisely identify the system's performance bottlenecks and perform targeted optimization.
  • More importantly, we can ascertain the maximum business capacity the platform can stably carry under the current system configuration. This data is the key basis for conducting scientific capacity planning and setting rate-limiting thresholds.

Emergency Drills: Forging "Combat Capability" in "Simulated Warfare"

Redundant design and stress testing are both preparing for the worst-case scenario. Emergency drills are about proactively and systematically causing the "worst-case scenario" to really happen within a controllable scope, in order to test whether our plans are effective, the team is proficient, and the tools are reliable. This is the ultimate means of improving system resilience and team emergency response capability.

Chaos Engineering

We introduce the concepts and tools of chaos engineering (such as Chaos Mesh), periodically and randomly injecting "minor faults" into the production environment and observing the system's response.

• Fault injection types:
  • Pod-level faults: Randomly kill a Pod of a certain service, testing whether it can be automatically restarted.
  • Node-level faults: Simulate the crash of a physical server (e.g., executing the reboot command), testing whether the services deployed on it can successfully drift to other nodes.
  • Network faults: Simulate network latency, packet loss, partitioning, etc., testing the application's performance and fault tolerance under weak network conditions.
• Learning from experiments: Every chaos experiment is a scientific exploration. If the system automatically recovers as expected, it verifies our high-availability design; if unexpected behavior occurs, it exposes the fragile links in our architecture, which need immediate repair and reinforcement.

Disaster Recovery (DR) Drills

This is an even larger-scale emergency drill than chaos engineering, aimed at simulating entire data center-level disasters.

• "Two Sites, Three Centers" architecture: For the highest-level core business, we plan a "active-active in the same city + remote disaster recovery" "Two Sites, Three Centers" architecture.
• Drill scenarios: We regularly (e.g., every six months or annually) organize a cross-data center switchover drill.
  • Tabletop exercise: All relevant teams (application, platform, network, storage, infrastructure) sit together and, based on a detailed drill script, step-by-step simulate the entire switchover process on paper, ensuring everyone is clear about their responsibilities and operation steps.
  • Real switchover drill: During a business low-traffic period, actually cut off the main data center's entry to the public network, or simulate a global "power outage" of the main data center, then activate the emergency plan, switch business traffic to the same-city or remote disaster recovery site, and verify that all core functions are normal.
• Quantify drill objectives:
  • Recovery Time Objective (RTO): The maximum tolerable time from disaster occurrence to service recovery at the disaster recovery site. Our goal is to continuously shorten RTO from hours to minutes through ongoing drills.
  • Recovery Point Objective (RPO): The maximum amount of data loss that can be tolerated after a disaster occurs. For core data, we need to use synchronous or near-synchronous data replication technologies to bring RPO close to zero.
• Review and improvement: After every drill, a in-depth review must be conducted to identify bottlenecks in the process, gaps in the toolchain, human operation errors, etc., and treat them as highest priority tasks for improvement.

Establish an Emergency Command and Coordination System

• Clear emergency organizational structure: We establish a "Computing Platform Emergency Command Center," led by company senior leaders and composed of heads from relevant departments, and an "Emergency Response Execution Team" composed of frontline technical experts.
• Standardized emergency response plans: We compile the "Computing Platform Overall Emergency Plan" and specialized plans for different disaster scenarios (such as power outage, fire, network paralysis, major security incidents). The plans clearly define information reporting paths, decision-making processes, command and coordination mechanisms, and external information release guidelines.
• Normalized training and awareness building: Through regular training, exams, and publicity, ensure that every operations and development personnel has basic emergency response awareness and skills.

By normalizing and institutionalizing the "combination" of redundant design, stress testing, and emergency drills, we are not just building a platform that "won't have problems"; we are building a platform that is "not afraid even if problems occur," with extremely strong anti-fragility. This resilience is the core capability that State Grid, as a critical infrastructure enterprise, must possess in the face of increasingly complex internal and external challenges.

Chapter Summary

In Chapter 8, we have drawn a solid and powerful conclusion to the entire technical support system of the computing platform. We started with problem localization and handling, establishing rapid response "tactical capability"; then through the health assessment system, we built the "diagnostic capability" of preventing problems before they occur; finally, through systematic high-availability assurance engineering, we forged the "strategic resilience" to withstand major risks.

At this point, all the content of Part 3, "Core Technical Support and Service Assurance," has been presented. From the infrastructure planning in Chapter 5, to the training platform support in Chapter 6, to the inference platform support in Chapter 7, and finally to the operations and high-availability assurance in this chapter, we have fully covered the full-lifecycle technical blueprint of the computing platform from "birth" to "growth" to "protection." This system is the technical confidence that allows us to confidently promise the business departments to provide "integrated, efficient, and easy-to-use" computing services, and it is the full engineering practice panorama of transforming the grand blueprint into a solid and reliable digital foundation. It proves that excellent computing services not only stem from advanced chips and algorithms but also from the relentless pursuit of technical details and the deep reverence for systems engineering.