祝融说。

If the first three chapters explored the top-level design, operations system, and strategic planning of computing construction — drawing the grand "battle map" and forming the fine "combat troops" — then starting from this chapter, we officially enter the "battlefield" of roaring artillery, going deep into the "trenches" of daily computing operations. This chapter focuses on the full life cycle management of computing resources, aiming to elaborate in detail how to implement the vision of "integration, high efficiency, and ease of use" into every resource request, every task schedule, and every capacity adjustment.

This is not only a technical issue but also a combination of management science and operational art. Its core goal is to build a data-driven, highly automated management closed loop from the demand entry point to the resource exit point. This closed loop, like a precise "resource circulation system," ensures that precious computing resources flow through the enterprise's "blood vessels" at high speed, with precision and without waste, delivering "nutrients" to the business "organs" that need them most. We will sequentially deconstruct the five key links of this closed loop: acceptance and estimation, application and allocation, dynamic adjustment, resource display, and resource recovery. These five links together compose a "symphony" of lean operations.

Demand Acceptance and Estimation: Scientific Assessment, Precise Calculation

The starting point of all operations comes from a clear, accurate demand. Demand acceptance and estimation is the "entry gate" of the computing resource life cycle and the "first valve" determining the efficiency and accuracy of all subsequent links. If this link is done roughly, allowing vague, exaggerated, and unreasonable demands to flood into the system, then the subsequent allocation, scheduling, and recovery will all be "building tall buildings on sand" — a lot of work with little result. Therefore, we must establish a set of scientific, rigorous assessment and estimation mechanisms here, achieving a transformation from "passive order-taking" to "active guidance, precise profiling."

Demand Acceptance: From "Free Market" to "Standardized Supermarket"

In the early stages of operations, demand acceptance is often chaotic. Users submit demands through various informal channels such as email, phone, and instant messaging, with inconsistent formats and incomplete information. Operations personnel spend a lot of time repeatedly communicating and confirming, which is inefficient and error-prone. Our primary task is to transform this "free market" into a "standardized supermarket" with a unified entry point, clear categories, and explicit rules.

Establish a Unified Computing Service Portal

We built a unified computing service portal for the entire company, the only legal entry point for all computing demand acceptance. This portal integrates multiple functions such as service catalog, demand application, status tracking, knowledge base, and monitoring dashboard, providing users with a one-stop service experience.

Design a Structured "Computing Demand Acceptance Form"

All demands submitted through the portal must fill out a structured "Computing Demand Acceptance Form." This form is meticulously designed to guide users in describing their demands in a standardized way, ensuring the completeness of key information.

• Basic information area: Applicant, department, project name, contact information, etc.
• Demand type selection area: Clearly differentiate between training computing power and inference computing power. This is the most important classification because the evaluation standards and resource models for the two are completely different.
  • Under training computing power, further subdivided into L1 large model training (long-term, large-scale, high parallel efficiency requirements), L2 model fine-tuning (short-term, medium-scale, fast turnaround requirements), and algorithm research and exploration (interactive, flexibility requirements).
  • Under inference computing power, further subdivided into online inference services (low latency, high concurrency requirements) and offline batch inference (high throughput requirements).
• Business background description area: Ask users to briefly describe the business objectives, expected value, and application scenarios of the computing demand. This helps the operations team understand the business context of the demand and evaluate its priority.
• Technical details quantification area: This is the core of the form, guiding users to quantify their technical demands.
  • For training computing power:
    • Model information: Model name/type (e.g., BERT, YOLOv5), framework (PyTorch/TensorFlow), parameter scale (e.g., 13B, 70B).
    • Data information: Training dataset size (GB/TB), data type (text/image/speech).
    • Resource requirements: Desired GPU model, number of GPU cards needed for a single training, expected single training duration, training frequency (e.g., once per week).
    • Software environment: Required CUDA version, Python version, core dependency libraries and versions.
  • For inference computing power:
    • Model information: Address of the model to be deployed, input/output format.
    • Performance target (SLA): Expected average latency, P99 latency, target QPS (peak and average).
    • Deployment mode: Whether high-availability deployment (multiple replicas) is needed, whether canary release capability is needed.

Through such a "foolproof" but informationally complete acceptance form, we have standardized the description of demands, freeing operations personnel from the tedious work of "piecing together information" and also forcing users to think more deeply before proposing demands.

Normalized Acceptance Mechanism and Preliminary Feedback

• Normalized acceptance: The service portal is open 7x24 hours, allowing users to submit demands at any time.
• Automated preliminary review: The system automatically checks whether the required fields of the acceptance form are complete. For applications with incomplete information, it directly rejects and prompts users to supplement.
• Timely response and feedback: After the ticket is generated, the system automatically sends a confirmation email to the applicant, informing them of the estimated evaluation cycle. The operations team (provincial L1/L2 or headquarters) will claim the demand within the specified SLA time (e.g., 1 business day) and conduct a manual preliminary review. For demands that are clearly unreasonable or have questionable information, they will proactively communicate with users for clarification.

Demand Estimation: From "Experience-Based Guesstimation" to "Scientific Modeling"

Demand acceptance solves the problem of "what it is," while demand estimation answers the difficult question of "how much is needed." This is the cornerstone of the entire resource allocation. Its precision directly determines resource utilization efficiency. We have abandoned the "guesstimation" model that relies on personal experience and established a set of scientific, standardized, and continuously iterative demand estimation system.

Establish Estimation Standard System and Methodology

Formulate a unified "Computing Demand Estimation Standard": This standard is the "constitution" of estimation work,clearly define the estimation methods, core indicator systems, and calculation formulas for different types of computing demands.

• Core indicator system:
  • Training core indicators: Model parameters, training data volume (Tokens/Samples), computation volume (FLOPs), effective hardware computing capacity (Effective FLOPs/s), communication overhead, memory footprint.
  • Inference core indicators: Computation volume per single request, model loading overhead, data pre/post-processing time, concurrency, batch size.
• Unified estimation methodology:
  • Training computing estimation: Adopt the "theoretical calculation + benchmark correction" method. First, calculate the total computation volume using FLOPs = 6PD (P for parameter count, D for training tokens — the commonly used GPT-3-style order-of-magnitude estimate), then divide by the effective computing power of the target hardware (measured through Benchmark, not theoretical peak) to get the theoretical training duration. Finally, based on network, I/O, and other overhead, multiply by a "correction factor" to obtain the final estimated duration.
  • Inference computing estimation: Adopt the "stress testing + capacity modeling" method. For new models, standardized stress testing must first be conducted to plot the "concurrency-latency-QPS" relationship curve. Then, based on the user's SLA requirements (e.g., latency < 200ms), find the maximum QPS a single card can support from the curve. Finally, based on total QPS demand and high availability strategy, calculate the total number of cards required.

Develop Standardized Estimation Models and Supporting Tools

To implement the complex estimation methodology, we developed a set of standardized computing demand estimation models (Excel templates or online calculators).

• Function: Users only need to input model parameters, data volume, target hardware, and other core parameters, and the tool automatically calculates the estimated training duration, resource requirements, and memory usage. For inference, input stress test data and SLA targets, and the tool automatically recommends the required number of GPU cards and configuration.
• Value:
  • Empower users for self-estimation: Open the tool to all users, helping them conduct scientific self-assessment before submitting demands, improving the quality of demand submissions.
  • Empower operations team review: The operations team uses the same tool to review user applications, making the review process evidence-based, fair, and transparent.

Headquarters and Provincial Estimation Implementation and Support

• Headquarters-level computing estimation implementation: For company-level major projects (such as "Guangming Electric Power" large model training) and cross-provincial resource demands, the headquarters intelligent computing operations team is directly responsible for estimation. They organize expert review meetings tovalidate the estimation plan, ensuring its scientific nature and reasonableness. The final "Computing Demand Estimation Report" serves as the decision-making basis for major resource allocation.
• Provincial computing estimation guidance and support: The headquarters is responsible for providing comprehensive estimation guidance to provincial units.
  • Formulate "Provincial Computing Estimation Work Guidelines," detailing the estimation process and standards.
  • Provide estimation tool templates adapted for provincial use.
  • Establish cross-regional experience exchange mechanisms, regularly organizing provincial operations teams to share estimation cases and experiences.
  • Provide expert support: For complex estimation problems encountered by provincial units, the headquarters provides remote or on-site expert support.

Build Supporting Assurance System

• Form a professional estimation team: At headquarters and in capable provincial companies, establish dedicated performance analysis and estimation positions, cultivating a group of composite experts who understand business, algorithms, and hardware.
• Compile standardized operation manuals: Compile the full estimation process steps, tool usage methods, common problems, etc., into detailed operation manuals.
• Build a unified estimation data management platform: Store the input parameters, process data, final results, and post-operation actual data of all previous estimations in a unified database. Through regression analysis of historical data, continuously iterate and optimize our estimation models, giving the estimation capability the ability to "self-learn and self-evolve."

Through this set of "standardized, tool-based, systematic" working mechanisms established in the demand acceptance and estimation link, we have successfully created a solid, reliable, and precise "entrance" for the full life cycle management of computing resources.

Resource Application and Allocation: Standardized Processes and Automated Delivery

After scientific estimation, demands enter the formal application and allocation phase. The goal of this phase is efficiency and fairness. We need to establish a set of standard, transparent, and efficient processes to ensure that every reasonable computing demand is met in the shortest possible time; at the same time, through automated technical means, minimize manual intervention, improve delivery speed and accuracy, and reduce operational costs.

Application Acceptance and Feasibility Assessment

After completing the estimation, users formally submit the "Computing Resource Application Form" containing the estimation results. Upon receiving the application, the operations team conducts a two-step assessment.

Preliminary Review of Completeness and Compliance of Application Materials

• Standardized entry: All applications must be submitted through the unified computing service portal, eliminating offline and informal application channels.
• Automated verification: The ITSM system automatically verifies whether the required fields of the application form are complete and whether key information (such as business demand description, model parameters, expected usage duration, etc.) is clearly filled in.
• Preliminary resource availability assessment: The system performs a preliminary feasibility assessment based on the current platform's real-time resource level. If the applied resource quantity far exceeds the current available surplus, the system automatically prompts the user and suggests adjusting the demand or accepting queuing.
• Compliance review: Operations personnel check whether the application complies with the company's computing usage specifications. For example, a non-core project in the exploration phase applying for a large number of high-performance, long-cycle strategically reserved resources may be non-compliant.

Multi-Dimensional Computing Feasibility Assessment

After passing the preliminary review, more professional technical personnel (possibly provincial L2 or headquarters architects) conduct an in-depth feasibility assessment.

• Resource suitability assessment:
  • Computing resources: Evaluate whether the applied GPU model and quantity match the model type (e.g., compute-intensive/memory-intensive) and parallel computing requirements. For example, for large-scale model parallelism requiring high-bandwidth NVLink, A100/H100 servers are recommended; for ordinary inference tasks, cost-effective T4 or A30 may be a better choice.
  • Storage resources: Evaluate whether the user's requirements for storage capacity, bandwidth, and IOPS match the storage types we provide (such as high-performance parallel file systems, SSD local disks, object storage).
  • Network resources: Evaluate the user's network bandwidth and latency requirements. Especially for distributed training tasks, ensure the allocated computing nodes are within the same high-speed interconnected network domain.
• Technical feasibility verification:
  • Hardware-software compatibility: Verify whether the software environment required by the user (such as specific versions of CUDA, PyTorch) is compatible with the platform's hardware and operating system.
  • Performance requirement matching: For inference services with strict SLAs, review whether the performance targets match the applied computing configuration — whether there is a case of "small horse pulling a big cart" or "big horse pulling a small cart."
  • Resource allocation conflict detection: In a multi-tenant environment, assess whether this resource allocation could negatively impact other users or high-priority tasks (such as network contention, storage I/O contention), and make necessary isolation or scheduling policy adjustments.

After the assessment, the technical personnel issue a brief assessment opinion,clearly state the conclusion of "approved," "suggest configuration adjustment," or "rejected," along with detailed reasons.

Approval and Allocation: Process-Driven and Precise Configuration

Application forms that pass the assessment enter the approval phase. We have designed a tiered approval process based on the importance of the demand and the amount of resources.

• Hierarchical approval:
  • Small/Standard resources: For standardized resource applications within a certain limit (e.g., fewer than 4 cards, usage less than 7 days), can be approved by the provincial operations manager or even automatically by the system, achieving "second-level" approval.
  • Large/Important resources: For resource applications exceeding the limit, with long occupation cycles, or involving core business, need to be routed to the headquarters operations manager or even department head for approval.
  • Strategic resources: For applications involving the headquarters' strategic reserve force resources, need to go through company-level expert committee review and management approval.
• Precise configuration and delivery:
  • Automated configuration: Once the approval takes effect, the ITSM system automatically completes precise resource configuration through API calls to the underlying cloud-native platform (Kubernetes). This includes creating the user's namespace, configuring resource quotas, applying network policies, mounting persistent volumes, etc. The entire process requires no manual backend login, achieving automated and standardized configuration.
  • Clear allocation parameters: The generated "Computing Resource Allocation List" clearly lists the resource quotas allocated to the user (e.g., CPU/Memory/GPU quantity), usage time limit (start time, end time), access credentials (e.g., Kubeconfig file, JupyterLab access link), and other key parameters.
  • Timely feedback and guidance: After resource allocation is complete, the system automatically sends a notification email to the applicant, informing them of the approval result, resource details, and providing links to the "Computing Usage Specifications" and user manual, guiding users to correctly and compliantly use the resources.

Emergency Response: Opening a "Green Channel" for Major Urgent Tasks

Standard processes ensure the order of daily operations, but we must also establish a "green channel" for "unplanned" major urgent tasks, reflecting the platform's responsiveness and assurance value.

• Define urgent tasks: Clearly define what level of tasks can activate the green channel, such as emergency analysis involving major grid safety accidents, research tackling national major strategic needs, repairing emergent models seriously affecting online business, etc.
• Simplify approval process: Applications for urgent tasks can bypass some routine approval steps, with rapid decision-making and authorization by the designated emergency responsible person (such as the headquarters operations director).
• Resource reservation and preemption mechanism:
  • Dynamic reservation: The platform always retains a portion of resources as a "dynamic reservation pool," not allocated to ordinary tasks.
  • Preemptive scheduling: When reserved resources are insufficient, the platform can, based on preset priority policies, temporarily "preempt" resources from running low-priority tasks (preempted tasks are gracefully terminated and their status saved, to be rescheduled after resources are released), ensuring the execution of high-priority urgent tasks.

By combining standard processes with emergency mechanisms, our resource application and allocation system both ensures orderly daily operations and possesses the resilience and flexibility to address emergencies.

Dynamic Adjustment: Elastic Scheduling Mechanisms for Training and Inference Computing

One-time resource allocation is not a permanent solution. Business loads change dynamically, and model requirements are continuously evolving. A rigid, static resource allocation model will inevitably lead to huge resource waste or service quality degradation. Therefore, establishing a set of elastic scheduling mechanisms that can dynamically adjust based on real-time conditions is the "advanced stage" of lean resource operations. We have designed different dynamic adjustment strategies for the different characteristics of training and inference scenarios.

Training Computing Dynamic Scheduling Mechanism: Balancing Assurance and Efficiency

Training tasks, especially large-scale distributed training, are time-consuming and have high interruption costs. The core goal of their dynamic scheduling is to maximize the overall throughput and resource utilization of the cluster while ensuring the timely completion of key tasks.

• Standardized cycle assurance plan: We have defined standardized assurance cycles for different types of training tasks. For example, for L2 fine-tuning computing power, we provide a "standardized 7-day training cycle assurance plan." Usersdefault get 7 days of exclusive usage rights, ensuring their fine-tuning task can be completed within a predictable cycle,free from interference from other tasks. This provides certainty for business agile iteration.
• Emergency demand rapid response channel: As mentioned earlier, open a green channel for major urgent training tasks, using preemptive scheduling to ensure the highest priority tasks immediately obtain the required resources.
• Dynamic scheduling based on real-time resource levels: This is key to improving cluster utilization. Our scheduling system (such as K8s scheduler based on Volcano or Yunikorn) has advanced scheduling capabilities:
  • Resource sharing and over-commitment: For research and exploration-type tasks with fluctuating resource demands, schedule them into the same resource pool and over-commit CPU and memory to a certain extent, using the principle of statistical multiplexing to improve resource utilization.
  • Job queuing and priority scheduling: When cluster resources are saturated, newly submitted tasks enter a queue. The scheduler dynamically adjusts the queue order based on multiple factors such as task priority, wait time, and resource demand quantity, ensuring high-priority and "small" tasks are not starved by "big" tasks.
  • Resource recovery and defragmentation: The scheduler regularly "defragments" resources released by preempted resources or short-duration tasks, recombining them into usable whole blocks for allocation to waiting tasks.

Inference Computing Intelligent Scaling Implementation: Pursuing Extreme Elasticity

Inference services face end users, and their loads often show obvious periodicity and burstiness. The core goal of their dynamic adjustment is to achieve on-demand resource usage while ensuring service SLA — i.e., "auto-scale during high load, auto-shrink during low load" — minimizing costs.

• Establish "dual-threshold" monitoring and early warning system: We set two core monitoring and early warning indicators and thresholds for each inference service as triggers for auto-scaling.
  • GPU utilization: For example, when a service's average GPU utilization exceeds 80% for the past 5 minutes, trigger scaling up; when it falls below 30%, trigger scaling down.
  • Memory usage rate: For some memory-intensive models, memory may become a bottleneck before the computing unit. We also set thresholds, such as continuously exceeding 90% triggering scale-up.
  • Why "dual thresholds": Monitoring both computing and memory dimensions simultaneously can more accurately capture resource bottlenecks, avoiding misjudgment from a single indicator.
• Formulate headquarters and province two-level collaborative expansion process:
  • Minute-level in-Pod expansion: When monitoring indicators trigger thresholds, the HPA (Horizontal Pod Autoscaler) mechanism deployed on Kubernetes automatically responds, increasing the service's replica count within minutes and distributing traffic to new replicas. This is the fastest and most routine expansion method.
  • Hour-level cross-Node expansion: If, after increasing replicas, the GPU node resources of the current cluster are completely used up, the provincial operations team receives an alert. They can manually or through automated scripts add new GPU nodes to the cluster from the provincial reserved resource pool, typically completed within 1 hour.
  • Day-level cross-province collaborative expansion: If all resources in the province are exhausted, the provincial team can initiate an "emergency resource support request" through the ITSM system to the headquarters. The headquarters operations team assesses the overall network resource status and allocates resources from other idle provinces or the headquarters resource pool for use by that province.
• Develop automated scaling tools: We have implemented a high degree of automation through a series of cloud-native toolchains such as Prometheus (monitoring), Alertmanager (alerting), and Kubernetes HPA/VPA (auto-scaling containers). Operations personnel only need to configure the scaling strategy (such as min/max replicas, trigger thresholds) when deploying the service, and the subsequent dynamic adjustment process is automatically completed by the platform.

Standardization of Computing Dynamic Adjustment Processes

To ensure that all dynamic adjustment operations are controllable and traceable, we have formulated the "Computing Dynamic Adjustment Implementation Specifications."

• Clearly define trigger conditions for various scenarios: Clearly define under what circumstances automatic scaling, manual adjustment, or preemptive scheduling can be triggered.
• Standardize approval processes: For major, manual adjustment operations, specify the required approval steps.
• Unified execution standard and recording requirements: All adjustment operations, whether automatic or manual, must be recorded in detail in the ITSM system or Configuration Management Database (CMDB), including the state before and after adjustment, operator/system, time, reason, etc., for audit and post-event analysis.

Through dynamic adjustment mechanisms tailored for training and inference scenarios, our computing platform has evolved from a "static resource provider" into an "elastic living organism" capable of adapting to business rhythms.

Resource Display: Transparent Management and Performance Publicity

If dynamic adjustment is the platform's "internal skill," then resource display is the platform's "external manifestation." Its core idea originates from the "publicity principle" in management science — regularly and transparently publicizing key operational data and performance indicators to all stakeholders (managers, users, operations team). This "display" behavior itself is a powerful management tool, bringing multiple benefits:

• Increase resource transparency: Break information asymmetry, allowing everyone to clearly see where resources are, how many there are, who is using them, and how they are being used.
• Build trust and fairness: Public data makes resource allocation and scheduling decisions more transparent, reducing user concerns about "backroom deals."
• Drive behavior optimization: When users see their resource utilization far below the average, they feel "peer pressure" and proactively optimize their applications.
• Provide decision basis: Provide the most direct and true data support for managers' capacity planning, investment decisions, and operational optimization.

Our resource display system consists of three parts: "real-time monitoring, regular publicity, and analysis and evaluation."

Real-Time Computing Usage Monitoring

This is the data foundation of display. Relying on the State Grid Cloud Platform and AI PaaS platform, we have established a comprehensive computing monitoring indicator system.

• Identify monitoring indicators:
  • Infrastructure layer: Server CPU/memory utilization, disk I/O, network bandwidth, power consumption, temperature.
  • Virtualization/Container layer: Pod resource consumption, GPU allocation and usage (collected through tools such as NVIDIA DCGM).
  • Application layer: Training task throughput, convergence speed; inference service QPS, latency.
• Build a unified monitoring platform: Aggregate all collected massive time-series data into monitoring systems such as Prometheus and VictoriaMetrics, and build multi-dimensional, interactive monitoring dashboards using tools such as Grafana.
• Scenario-oriented periodic reports:
  • Training computing power: Automatically generate the "Training Computing Resource Usage Report" on a per-"model training round" basis. The report details the minute-level utilization curves of GPU/memory/network, performance bottleneck analysis, etc., providing refined optimization references for model developers.
  • Inference computing power: Automatically compile the "Inference Computing Resource Usage Report" on a monthly basis, summarizing the average/peak QPS, latency distribution, resource consumption, etc., of the services for that month, providing raw data for computing usage evaluation.

Computing Usage Publicity

We aggregate, anonymize, and summarize monitored data, regularly publicizing it to the entire company in standardized form.

• Publicity cycle: Monthly as the main cycle.
• Publicity channels: Computing service portal homepage, i State Grid news, monthly operations email.
• Publicity content ("Computing Resource Display Report"):
  • Computing resource levels: Clearly display the company's total computing scale, allocated scale, available surplus, and the quantity and distribution of various GPU models.
  • Global operations health: Show the previous month's network-wide average computing utilization, task queuing time, number of major faults, and other macro indicators.
• "Red and Black List" — Resource usage efficiency ranking:
  • Red list: Publicize the users or projects with the "highest resource utilization," "best model performance," and "lowest unit cost" from the previous month, commending and promoting them.
  • Black list: Publicize the users or projects with the "lowest resource utilization," "longest idle time," and "largest application-usage deviation," issuing named warnings.
  • Computing usage economic indicators: Analyze the "duration-performance-power consumption" relationship of typical model training, guiding users to focus on the comprehensive cost of computing usage.
  • Performance indicator TOP list: Show the applications with the highest CPU, GPU, memory, and utilization rates, sharing their optimization experiences.

Computing Usage Analysis and Evaluation

Publicizing data is not the endpoint; more important is conducting in-depth analysis and evaluation based on the data.

• Compile "Computing Resource Usage Analysis and Evaluation Report": The operations team writes a more in-depth analysis report each month based on the displayed data.
• Trend analysis: Analyze the changing trends of key indicators such as total computing usage and utilization, comparing with historical data from the same period.
• Anomaly warning and diagnosis:
  • Low-efficiency warning: Issue a "yellow" warning for training and inference computing resources with monthly average utilization and memory usage below 20%, and proactively contact users to analyze the reasons.
  • Overload warning: Issue an "orange" warning for resources with monthly average utilization above 80%, indicating potential performance bottlenecks and capacity risks, and coordinate with the capacity planning team for assessment.
  • Thematic analysis: For typical problems that emerged in the month (such as network congestion in a certain cluster, universally low training efficiency for a certain type of model), conduct thematic drill-down analysis and propose improvement suggestions.
  • Annual summary and planning input: At the end of each year, the operations team compiles the "Annual Computing Usage Summary," comprehensively evaluating the configuration, management, and usage level of computing resources for the year, summarizing experiences and lessons. This summary report is one of the most important inputs for the next year's computing consolidated planning and operational optimization.

Through the "display" combination, we have successfully made the originally "invisible and intangible" computing usage status "visible to all under the sunlight," using the power of transparency to drive thespiral upward of management level and resource efficiency.

Resource Recovery: Improving Turnover Rate, Eliminating Resource Waste

If the preceding four links are about how to "better allocate and use" resources, then resource recovery is about how to "gracefully end and reuse." It is the "last mile" of the full resource life cycle management and the key link to ensure that the entire resource circulation system does not "block." A computing platform lacking an effective recovery mechanism will eventually have its resource pool filled with a large number of "zombie resources," leading to the dilemma of "having resources but being unable to use them." Our recovery mechanism follows the principles of "categorized policy, proactive management, user-friendly."

Training Computing Recovery: Combination of Periodic and Expiration-Based Systems

The use of training computing power has distinct phases, so recovery strategies need to balance the different needs of long-term research and short-term tasks.

• Refine recovery standards, categorized management:
  • L1 large model training computing power: Periodic assessment recovery
    • Resource division: At the time of application, resources are divided into dedicated pools for semantic, multimodal and visual, and scientific computing models, avoiding resource contention between different types of large models.
    • No fixed usage period: Considering the long-term nature of large model training and iteration, no fixed usage deadline is set.
    • Based on "Guangming Electric Power" large model iteration needs: Each month, the headquarters large model project team and computing operations team jointly assess whether the computing resources allocated to each model direction are fully and efficiently utilized and whether phased iteration goals have been met. For those with slow progress and low efficiency, proposals to recover some resources and redistribute them to more efficient teams can be made. This is an output-based dynamic assessment recovery mechanism.
  • L2 model fine-tuning computing power: Automatic recovery upon expiration
    • Clear application period: When users apply for fine-tuning computing power, they must specify the usage period, and in principle, no more than 7 days.
    • Automatic recovery and redistribution upon expiration: After the fine-tuning task ends or the 7-day period expires, the system automatically executes recovery operations, releasing resources back to the public resource pool for redistribution. This ensures that resources for agile iteration achieve "fast borrowing and fast returning," with very high turnover rate.
    • User-friendly extension application mechanism: We allow users to submit an extension application through the ITSM process before resource expiration (e.g., 3 business days). The application must detail the reason for the extension and the required extension time. The operations team reviews based on the reasonableness and the current resource level of scarcity. If no extension is needed, recovery proceeds on time.

Inference Computing Recovery: Usage Rate as Primary, On-Demand as Secondary

Inference computing power is a long-term online service, not suitable for the "expiration recovery" model. The core of its recovery strategy is to ensure that allocated resources match the actual load they carry, and redundant resources are recovered promptly.

• Formulate computing recovery mechanism: Regular recovery as primary, on-demand recovery as secondary
  • No fixed usage time limit: As long as the service is online and has reasonable access volume, its resources will not be recovered.
  • GPU and memory utilization as primary standard: This is the main basis for regular recovery. Our automated monitoring system pulls resource usage reports for all online inference services every month. For those services where the daily peak GPU utilization and memory utilization have both been consistently below 20% for the past month, the system automatically marks them as "pending recovery candidates."
  • Monthly assessment and communication: The operations team reviews this "candidate list" and proactively communicates with the responsible persons of related services to verify the reasons for low load (is it because the business is in early stages with low traffic, or has the business gone offline and been forgotten?). If confirmed as redundant resources, a recovery plan is negotiated.
• On-demand recovery of redundant computing power:
  • Model change or shutdown-driven recovery: When a business system upgrades its version, replacing the original AI model (e.g., replacing a large model with a smaller one), or when an entire AI functional module is taken offline, the business responsible person has the responsibility to proactively submit a "Computing Resource Unsubscription Application" through the service portal, returning the no-longer-needed redundant computing power. In our assessment mechanism, "proactive unsubscription" is counted as a bonus item.
  • Platform-initiated on-demand recovery: In addition to the above regular assessment, if the operations team discovers through monitoring that a service's required computing power has significantly decreased due to model optimization, business traffic decline, etc., they can also proactively initiate the recovery process, negotiating with the user to recover the redundant portion.

Standardization and Automation of Recovery Processes

All recovery operations must follow standard processes and be automated as much as possible to reduce human errors and disputes.

• Generate "Computing Resource Recovery List": Regardless of the recovery scenario, a "Computing Resource Recovery List" is ultimately generated, clearly recording the details of the recovered resource, the reason for recovery, the original using unit, recovery time, etc.
• Notification and confirmation before recovery: Before executing the recovery operation, the system automatically sends multiple notifications to the user (e.g., 7 days, 3 days, 1 day in advance), giving them time to back up data and handle applications. For forced recovery, there must be clear approval records.
• Automated recovery execution: Recovery operations are automatically executed by the platform through APIs, including deleting related deployments, services, releasing storage volumes (PVCs), clearing user configurations, etc.
• Post-recovery audit and reporting: After recovery is complete, relevant records are archived, and the monthly operations report summarizes and analyzes the recovery activities for the month, as part of operational performance.

Chapter Summary

In Chapter 4, we have, like a skilled butcher dissecting an ox, finely decomposed the seemingly monolithic computing resource operations into five interlocking practical links: acceptance and estimation, application and allocation, dynamic adjustment, resource display, and resource recovery. We have seen that by introducing the concepts of "standardization, automation, and data-driven" in each link, we have successfully built an efficient, transparent, and lean full life cycle resource management closed loop. This closed loop is not only the guarantee for the daily stable operation of the computing platform but also the core engine for realizing the "high efficiency" operational vision. It ensures that the nation's huge investment can be transformed into a continuous stream of highly utilized, truly business-empowering "new era electricity."