祝融说。

In Chapter 5, we built a rock-solid infrastructure for the computing platform. This is like having constructed a wide, smooth F1 track equipped with a top-tier monitoring and dispatch center. Now, it is time for the true protagonist — the "training race car" carrying the mission of model iteration and innovation — to hit the track. Model training is the source of AI value creation, and also the link with the greatest computing power consumption and the most densely packed technical challenges. The training process for a large model with hundreds of billions or even trillions of parameters often requires hundreds or thousands of GPU cards to work together for weeks or even months. Any oversight in any link can lead to "one careless move and the whole game is lost" — massive computing costs sunk, precious R&D time windows missed.

Therefore, building a professional, in-depth, comprehensive training platform technical support system is the key "catalyst" and "multiplier" for transforming computing resources into model capabilities. This chapter focuses on the technical support for the training link, revolving around three core issues:

1. Building and Tuning the "Car": How to efficiently deploy, integrate, and optimize a large-scale training cluster, ensuring it has the best performance baseline from birth? (6.1 Deployment, Integration, and Optimization of Training Clusters)
2. Full-Process Escort for the "Race": During the long and fragile training process, how to provide high-reliability assurance for key training tasks, minimizing Mean Time to Recovery (MTTR) and ensuring the successful completion of the "race"? (6.2 High-Reliability Assurance for Key Training Tasks)
3. Skill Improvement for the "Driver": How to help users better adapt their training models to our platform and perform in-depth performance tuning, enabling the "driver" to squeeze the ultimate performance out of the "race car"? (6.3 Model Adaptation, Migration, and Performance Tuning)

Through systematic answers to these three questions, we will fully present a three-dimensional training technology support matrix spanning from hardware to software, from platform to model, and from performance to stability.

Deployment, Integration, and Optimization of Training Clusters

A high-performance training cluster cannot be automatically achieved by simply connecting a bunch of AI servers together. It is a complex system with tight hardware-software coupling. The three subsystems of computing, storage, and network must be precisely "integrated" and "optimized" like different sections of an orchestra, playing in harmony to produce a magnificent performance. Any section "going off-key" or "dropping the chain" will become the weak link of the entire orchestra's performance. This section details how we ensure every newly built or expanded training cluster reaches a "peak performance straight out of the factory" state through systematic engineering methods.

Training Cluster Integration: Precise Assembly from "Parts" to "Complete Vehicle"

Integration is the final stage of cluster deployment and the most critical "last kick." Its goal is to verify whether the various independent subsystems (computing, storage, network) and the AI platform software stack can work together as a whole after being combined, meeting design expectations.

Inter-Subsystem Integration of Computing, Storage, and Network

• Computing-Storage Integration: Verify whether computing nodes can correctly and efficiently mount and read/write storage resources. We conduct benchmark tests, such as using fio or iozone tools to perform large file read/write tests on the shared file system concurrently from multiple computing nodes, ensuring its aggregate bandwidth and IOPS meet the requirements for large-scale data loading. We simulate real data loading scenarios to troubleshoot possible mount permission issues and read/write performance degradation caused by network jitter.
• Computing-Network Integration: This is the top priority of the integration. The core is to verify communication performance between GPUs.
  • Intra-node P2P communication test: Use tools such as p2pBandwidthLatencyTest to test the point-to-point (P2P) bandwidth and latency between all GPU cards within the same server, verifying whether NVLink/NVSwitch or PCIe links are functioning correctly.
  • Cross-node RDMA communication test: Use tools such as ib_write_bw and perftest to test the RDMA (Remote Direct Memory Access) performance between any two computing nodes through RoCE or InfiniBand networks, ensuring network configurations (such as PFC, ECN) are correct and can achieve line-rate performance.
  • NCCL All-Reduce Performance Benchmark: Use the nccl-tests tool to run All-Reduce benchmark tests on different scales (2, 4, 8... full nodes). This is the test that best simulates the communication load of real distributed training, and its results (Bus Bandwidth) are the "gold standard" for measuring the parallel computing efficiency of the entire cluster. We compare the test results against NVIDIA's official benchmarks for similar hardware configurations. Any significant gap indicates network or system configuration problems.

AI Platform Interface Integration

• Resource scheduling integration: Verify whether the upper-layer Kubernetes + Volcano/Yunikorn scheduler can correctly identify and schedule underlying GPU resources. We conduct a series of scheduling tests, such as submitting an MPI task requiring 8 GPU cards, verifying whether the scheduler can find a complete 8-card node to run it; or submitting a task requiring cross-node operation, verifying whether Gang Scheduling can ensure all Pods are started simultaneously.
• Monitoring system integration: Verify whether monitoring components (such as Prometheus, DCGM-Exporter) can successfully collect "computing-storage-network" indicators from all nodes and correctly display them on the Grafana dashboard.
• Storage interface integration: Verify whether the AI platform can dynamically create and mount the required storage volumes for training tasks through the CSI (Container Storage Interface) plugin.
• Output: "Training Cluster Integration Report"
  • This report details all integration test items, methods, process data, and final results, comparing them with design objectives. For items not meeting standards, detailed root cause analysis and solutions are attached. Only when all integration items pass testing is the cluster considered to have the basic conditions for "delivery and launch."

Training Cluster Indicator Design: Establishing a "Standard of Measurement" for "Performance"

Integration ensures the cluster is "usable." To measure whether it is "good to use," a scientific, quantifiable indicator system is needed. The indicator system we designed aims to comprehensively evaluate the overall capability of a training cluster from multiple dimensions.

• Cluster computing scale indicators:
  • Total theoretical computing power (TFLOPS): The sum of the theoretical peak computing power (e.g., FP16 or BF16) of all GPU cards in the cluster. This is the basic indicator measuring how much "muscle" the cluster has.
  • Effective total computing power (EFLOPS): Obtained by running standard hybrid-precision Linpack benchmarks such as HPL-AI, this reflects the cluster's effective computing power in actual operation. This indicator better reflects the cluster's true computing capability.
• Cluster utilization indicators:
  • Physical utilization: The proportion of allocated GPU cards to the total number of cards in the cluster.
  • Logical utilization: The average usage rate of the actual computing units among the allocated GPU cards. High physical utilization with low logical utilization is a typical sign of "occupying the toilet without shitting," a key focus of operational optimization.
• Cluster stability indicators:
  • Mean Time Between Failures (MTBF): The average time a cluster can continuously run without major faults.
  • Node failure rate: The proportion of computing nodes with hardware or software failures per unit time.
• Cluster parallel efficiency indicators:
  • Scaling efficiency: When using N nodes for distributed training, the speedup ratio of training speed compared to using a single node, divided by N. Ideally, scaling efficiency should be close to 100%. This indicator is the core measure of the cluster's network architecture and communication efficiency.
• Output: "Training Cluster Indicator Design Report"
  • This report defines the precise calculation formulas, data collection sources, and display methods for the above indicators, making them the "common language" for all subsequent performance testing and optimization.

Training Cluster Monitoring: Using "Data" to See Through "Health"

Based on the designed indicator system, we built a unified cluster monitoring platform for continuous, in-depth "physical examinations" of the cluster.

• Unified monitoring: Aggregate all indicator data into a unified monitoring platform and create a "Training Cluster Health" dedicated dashboard through Grafana, achieving centralized viewing and unified management of all cluster states.
• Training cluster stress testing:
  • Test object: We select the company's typical large model (such as the "Guangming Electric Power" semantic model) as the test object because it best represents real high-load scenarios.
  • Test method: Under different cluster scales (such as 32, 64, 128 nodes), conduct "pull tests." We run the model at full load for a period, monitoring whether its various functions (such as checkpointing, logging) are normal and whether performance indicators (such as throughput, scaling efficiency) meet expected baselines.
• Output: "Training Cluster Stress Test Report," used to verify the cluster's performance and stability under high load.
• Training cluster long-duration stability test:
  • Test method: Use open-source, industry-recognized stable models (such as GPT-3, LLaMA) to conduct continuous training for several days or even a week on multiple newly built training clusters. This process is like "burning in," aiming to expose those sporadic hardware and software problems that are difficult to find in short-term tests (such as memory leaks, network transient disconnection, GPU overheating throttling, etc.).
  • Stability optimization: Any problems found during the long-duration stability test are recorded, analyzed, and pushed vendors or internal teams to fix and optimize until the cluster can pass a 7x24 hour stability test.
• Output: "Training Cluster Stability Test Report," the final "pass" for the cluster to carry critical training tasks.

Training Cluster Tuning: The Art from "Can Run" to "Run Fast"

Having passed all the above tests, the cluster already has a good foundation. However, to maximize its performance, a series of refined tuning efforts are needed. This is like an F1 car needing personalized setup based on the track and driver characteristics after leaving the factory.

Training Cluster Performance Bottleneck Analysis

• Data-driven bottleneck identification:
  • Establish performance baseline: First, run a series of standard Benchmark programs to establish the cluster's performance baseline in its "bare" state (such as memory bandwidth, network latency, storage IOPS, etc.).
  • Profile real business loads: Select a typical business training task and use tools such as NVIDIA Nsight to conduct end-to-end performance profiling. The profiling results clearly show the detailed time distribution within a training iteration: how much time is spent on data loading (I/O), CPU preprocessing, GPU computation, and inter-node communication.
  • Quantify bottleneck impact: Through the profiling chart, we can clearly identify the core reason for low GPU utilization (so-called "GPU starvation") and quantify its impact. For example, "30% of the time, the GPU is waiting for the CPU to complete data preprocessing."

Training Cluster Scheduling Optimization

• Scheduling strategies for different businesses:
  • Topology-aware scheduling: For large-scale training tasks requiring high-bandwidth communication, we configure the scheduler to be "topology-aware." The scheduler preferentially assigns all Pods of a task to the same switch, or even the same rack, minimizing cross-switch communication latency.
  • Resource preemption and fair sharing: For R&D clusters shared by multiple small users, we introduce more refined fair-sharing scheduling strategies (such as DRF - Dominant Resource Fairness), ensuring each user gets a relatively fair share of resources, preventing any single "big user" from occupying all resources for a long time.

Training Cluster Optimization Effect Evaluation

• Quantify optimization results: All tuning operations must be measured against data. We conduct A/B testing before and after optimization to quantify the improvement in key indicators.
  • Resource efficiency dimension: How many percentage points did the average GPU utilization increase?
  • Performance improvement dimension: How much did the training throughput of typical models increase? How many hours was the end-to-end training duration reduced?
  • Cost savings dimension: How much did the total computing cost (card-hours) consumed for completing the same training task decrease?
• Output:
  • "Training Cluster Resource Optimization Service Plan": Details the bottleneck analysis process, specific tuning measures, and operating steps.
  • "Training Cluster Optimization Effect Evaluation Report": Uses data and charts to clearly demonstrate the value brought by optimization, providing data support for continuous improvement and precipitate as best practices in the knowledge base.

Through this systematic engineering approach from deployment and integration to optimization, we ensure that the infrastructure of every training cluster reaches its best state, providing a solid, efficient, and reliable "starting platform" for the training tasks carried on top.

High-Reliability Assurance for Key Training Tasks (MTTR Management)

A ten-thousand-card-level large model training task lasting months is more fragile than one might imagine. Over such a long timescale and at such a large cluster scale, hardware failures (GPU cards, servers, switches) are almost inevitable events. Every interruption means valuable computing time is wasted. Therefore, simply pursuing high performance is far from enough; we must place high reliability on an equal footing.

We introduce MTTR (Mean Time To Recovery) as the core operational indicator for measuring the assurance capability of training tasks. The shorter the MTTR, the stronger our ability to recover training from faults, and the smaller the impact on training efficiency. Our assurance system builds a full-process MTTR management closed loop around three stages: "pre-training, during-training, and post-training."

Pre-Training: Conduct Cluster Health Checks, Prevent Problems Before They Occur

The best fault handling is to prevent faults from happening. Before the start of any key training task (especially large-scale, long-cycle large model training), we conduct a "military exercise" level, in-depth cluster health check.

• In-depth hazard investigation:
  • Hardware stress testing: Use specialized tools (such as gpu-burn, stress-ng) to conduct sustained, multi-hour full-load stress testing on all computing nodes that will participate in training, proactively exposing unstable hardware components that arenot easily discovered under low load (such as poorly binned GPUs, improperly cooled CPUs).
  • Comprehensive network check: Re-perform point-to-point bandwidth and latency tests on all participating nodes' network links, ensuring no "sub-healthy" link has slipped into the cluster.
  • Firmware and driver version consistency check: Automated scripts scan all nodes to ensure that the versions of BIOS, BMC, NIC, GPU firmware, and drivers are completely consistent, eliminating inexplicable problems caused by version mismatches.
• Quick check and diagnostic stress test:
  • Pre-training task quick check: At the last moment before task startup, run a lightweight, standardized "quick check script" to confirm the cluster's core state one final time within minutes.
  • Fault diagnosis stress test: Simulate some common fault scenarios (such as manually killing a Pod, pulling out a network cable) to verify the platform's fault detection and self-healing capabilities (such as automatic Pod restart) meet expectations.
  • Resource pool change stress test: If node replacement or expansion is planned during the training process, pre-run and stress test the change operation to ensure it won't impact the running task.
• Output: "Training Cluster Check Report"
  • This report lists the results of all check items. Only when all check items show "passed" and there are no "warning" or "risk" items do we formally approve the launch of this key training task. This "check first, play later" model greatly avoids preventable hardware and software problems.

During Training: Rapid Fault Perception and Recovery in Clusters

Despite meticulous prevention, faults may still occur during long-cycle training. At this point, the ability for rapid "perception-localization-isolation-recovery" is key to shortening MTTR.

• Rapid perception:
  • Multi-dimensional monitoring: Our monitoring system monitors multiple key indicators of the training task in real time.
    • Task logs: Through log aggregation systems such as Loki, monitor the output logs of the training task in real time and set keyword alerts (such as Error, Exception, NCCL timeout).
    • Core performance indicators: Monitor training progress (Steps/sec), Loss value, average GPU computing power, etc. in real time. An abnormal drop or stall in any indicator may be a sign of a fault.
  • Intelligent Alerting: The alerting system automatically correlates information from different sources (logs, metrics, hardware monitoring) for intelligent noise reduction and aggregation, sending precise alerts to operations personnel at the first moment.
• Rapid localization:
  • Log analysis and correlation: When an alert occurs, operations personnel can quickly drill down through the unified monitoring platform to the task logs, system logs, DCGM metrics, and network telemetry data at the time of the anomaly, conducting cross-domain correlation analysis. For example, discovering that the training Loss is not converging while simultaneously observing an abnormal increase in ECC error count on a particular GPU card allows preliminary localization to that GPU card's hardware problem.
  • Fault knowledge base: We distill the experience from past fault handling into the "Training Cluster Fault Handling Knowledge Base." When a similar alert or log pattern is encountered, historical solutions can be quickly retrieved.
• Rapid recovery:
  • Hardware problem rapid isolation: Once a problem is localized to a specific hardware component (such as a GPU card, server), our primary principle is "recover training first, investigate hardware later."
    • Hot swap/hot plug: For hot-swappable components, perform online replacement directly.
    • Node isolation and replacement: If an entire server fails, the operations personnel immediately mark the faulty node as "unschedulable" and allocate a healthy node from the spare resource pool to join the training (requiring the training framework to support elastic scaling).
  • Software problem rapid rollback: If the fault is caused by software upgrades or configuration changes, immediately execute the contingency plan and roll back to the last stable version.
  • Linkage with the training framework: Modern deep learning frameworks (such as PyTorch FSDP) have good fault tolerance. As long as the number of failed nodes does not exceed a certain threshold, the framework itself can automatically reduce the training scale and continue running after losing some nodes, or automatically recover from the nearest checkpoint. Our task is to quickly supplement healthy nodes to restore training to its maximum scale.
• Output: "Training Cluster Fault Handling Report"
  • Every fault handling process must be recorded in detail, including the fault phenomenon, localization process, solution, and recovery time (MTTR).

Post-Training: Summary Analysis and Continuous Optimization of Cluster Faults

Every fault is a valuable learning opportunity. After a key training task ends, we organize a dedicated review meeting to summarize and analyze all faults that occurred during training.

• Analyze fault patterns: Categorize and statistically analyze all "Fault Handling Reports," identifying the highest-frequency and most impactful fault modes (Failure Mode). For example, we might find that "a certain batch of GPU cards shows a significantly increased probability of memory instability after running for over 1000 hours."
• Optimize fault handling processes: Based on the review results, revise and optimize existing fault handling plans and operation manuals, making them more targeted and actionable.
• Drive platform and hardware improvements: Feedback the analyzed systemic hardware or software defects to vendors, pushing them to fix them in the next product generation or the next software version.
• Distill training task assurance knowledge base: Systematically distill the results of summary analysis into the knowledge base, including new fault modes, better diagnostic methods, faster recovery processes, etc., forming a closed loop of knowledge and capability iteration.
• Output: "Training Cluster Assurance Summary Report"
  • This report is the final distillation of the entire key task assurance effort and the "roadmap" for us to continuously reduce MTTR and improve cluster availability.

Through this full-process high-reliability assurance system of "pre-event inspection, in-event rapid response, post-event review," we have transformed large-scale training from a high-risk activity "left to fate" into a deterministic engineering effort "under control," providing the most solid stability assurance for the company's core AI R&D tasks.

Model Adaptation, Migration, and Performance Tuning

The platform and cluster are the "stage," and the model is the true "dancer" on the stage. No matter how good the stage, if the dancer is not suited to it or has poor technique, a wonderful performance cannot be delivered. The service of adaptation, migration, and tuning of training models is our core work as "stage technical directors," helping the "dancer" perform at their best. This service aims to solve three types of problems users encounter in model development and training: the model "cannot run" on our platform (adaptation), the model "suffers from acclimatization" after migrating from another platform (migration), and the model can run but "runs too slowly" (tuning).

Training Model Computing Demand Assessment and Adaptation Risk Estimation

Before the model officially goes on the platform for training, conducting an in-depth "physical examination" is the key first step to avoid a lot of subsequent rework and problems.

• Assess existing and required computing power:
  • We work with users to precisely assess the model's computing demands (compute, memory, communication) using the estimation tools introduced in Chapter 4.
  • At the same time, we assess the computing resource quotas the user currently has (or plans to apply for), analyzing the "demand-supply" match.
• Confirm the target platform's support capabilities:
  • Hardware compatibility check: Confirm whether the computation precision requirements of the model (such as FP32, TF32, FP16, BF16) are supported by the target GPU hardware.
  • Software stack compatibility check: Comprehensively check whether the versions of core libraries (CUDA, cuDNN, NCCL, PyTorch, etc.) that the user's code depends on are compatible with the standard environment provided by our platform. For incompatible parts, assess the feasibility and workload of upgrading or downgrading.
• Estimate adaptation risks in advance:
  • Custom Operators: Check whether the user's model code contains a large number of custom C++/CUDA operators that depend on specific hardware platforms or CUDA versions. This is the most common "pitfall" in model migration because these operators often need to be recompiled or even rewritten.
  • Private or closed-source dependencies: Check for dependencies on private code libraries or closed-source software, which may bring licensing and compatibility issues.
• Output: "Training Model Computing Adaptation Assessment Plan"
  • This plan gives a clear conclusion: whether the model is "directly runnable," "needs minor adaptation," or "has significant adaptation risks." For models needing adaptation, the plan includes a detailed adaptation roadmap, estimated workload, and responsibility division.

Model Adaptation and Migration Service: Clearing the Obstacles of "Acclimatization"

For models that require adaptation and migration after assessment, our technical support team provides expert-level "accompaniment" services.

• Formulate migration adaptation path: Based on the assessment report, formulate a detailed, step-by-step migration adaptation plan, ensuring the entire process is orderly and controllable.
• Provide code modification support:
  • Environment dependency adaptation: Assist users in packaging their code into standard Docker images, solving environment dependency problems.
  • Custom operator refactoring and development: For the most thorny custom operator problems, our experts work with users to refactor the code so that it can successfully compile and run under the new platform's CUDA version. For key performance bottleneck operators, we can even provide high-performance operator custom development services.
  • Model structure adjustment: When necessary, suggest users make minor adjustments to parts of the model's network structure to better adapt to the computing characteristics of the target hardware (e.g., using convolution kernel sizes that can be accelerated by Tensor Cores).
  • Maintain model accuracy: Throughout all adaptation modifications, we adhere to "maintaining the original model accuracy" as the first principle. We establish a benchmark testing process, performing regression testing on model accuracy at every key node of the adaptation work, ensuring no unintended negative impact is introduced.

Training Model Tuning: Squeezing Every Drop of Performance

After the model can successfully run, the performance tuning curtain has just risen. Our goal is to, through systematic methods, push the model's training efficiency to the limit.

• Resource usage analysis and bottleneck identification: This is the starting point of tuning. We again use profiling tools such as NVIDIA Nsight to conduct in-depth analysis of the model's performance under real training loads, precisely identifying whether the bottleneck is compute-bound, memory-bound, or I/O-bound.
• Provide full-stack performance tuning services:
  • Data pipeline optimization:
    • Guide users to use efficient data loading libraries (such as DALI, FFCV).
    • Optimize data preprocessing logic, offloading it from the CPU to the GPU.
    • Enable multi-process data loading (num_workers) and enable memory pinning (pin_memory), ensuring data can be quickly fed into the GPU.
  • Mixed Precision Training: Guide users to enable AMP (Automatic Mixed Precision), using the GPU's Tensor Core units to significantly increase training speed and reduce memory usage with almost no loss of accuracy (the magnitude of improvement varies by model and hardware; typically around 2x in typical scenarios, an empirical range rather than a fixed promise).
  • Distributed training strategy optimization:
    • Based on the model's characteristics and cluster network conditions, recommend the optimal combination of parallelism strategies for the user. For example, for models with huge parameter counts but small activation values, recommend ZeRO-3 (an advanced data parallelism) + tensor parallelism; for very deep models, recommend introducing pipeline parallelism.
    • We assist users in modifying their training code to support these advanced parallelism strategies.
  • Gradient Accumulation: Guide users to use gradient accumulation technology when memory is insufficient to support a large Batch Size, achieving equivalent large-batch training.
  • Operator Fusion and Compilation Optimization: Guide users to use PyTorch 2.0's torch.compile() or NVIDIA's Apex tools to automatically fuse multiple small operators into one large GPU Kernel, reducing Kernel Launch overhead and improving computing efficiency.
  • Checkpointing Optimization: For large models requiring frequent checkpoint saving, guide users to use asynchronous, distributed checkpointing schemes, reducing its blocking of the main training pipeline.
• Output: "Training Model Tuning Plan"
  • This plan is like a detailed "physical examination report + fitness plan," clearly listing the model's performance bottlenecks, specific optimization suggestions (with code examples), and expected performance improvement targets.

Post-Adaptation Model Verification and Deployment Implementation Services

• Adaptation migration verification: After completing all adaptation and tuning work, conduct a formal, end-to-end verification test.
  • Function and core performance indicator evaluation: On standard datasets, perform final verification of the model's core functions and accuracy indicators, ensuring they are completely consistent with pre-migration levels.
  • Stability and concurrency stress testing: Conduct long-duration stability testing and simulate multi-user concurrent submission of training tasks, verifying the model's stability under complex loads.
• Output: "Training Model Adaptation Migration Verification Report"
• Training model deployment implementation services:
  • Compile "Training Model Deployment Implementation Plan": Package the entire adapted and tuned model, code, environment, and best practice parameters into a standardized "Training Solution Package."
  • Provide deployment implementation technical support: Assist users in deploying and promoting this solution package at scale under the target computing resources, ensuring other teams or projects can "one-click replicate" the best training results.
• Output: "Model Deployment Implementation Report"

Chapter Summary

In Chapter 6, we have deeply explored the "uncharted territory" of training platform technical support. Through cluster deployment, integration, and optimization, we ensured the "stage" was solid and efficient; through high-reliability assurance for key tasks, we provided worry-free "escort" for the "performance"; and finally, through model adaptation, migration, and performance tuning, we helped the "dancer" to show their best "dance moves." This series of interlocking, progressively deepening technical support services together constitutes a powerful empowerment system. It not only solves the various technical problems users may encounter in model training but, more importantly, it transfers the company's top-tier engineering optimization capabilities to every AI developer in the form of services, truly achieving "letting professionals do professional work" — allowing algorithm scientists to focus on algorithm innovation and business experts to focus on scenario mining, thereby maximizing the acceleration of AI innovation and implementation within the enterprise.