祝融说。

In the previous two parts, we systematically elaborated on the top-level design, operations system, and strategic planning of computing platform construction. Together, these form the "soft power" of platform construction — the scientific management framework and forward-looking strategic layout. However, advanced concepts ultimately need to be carried by technological excellence. Starting from this part, we will delve into the technical "hard core" of the computing platform, fully deconstructing the core technical support and service assurance systems that underpin the platform's efficient and stable operation. This is the concentrated embodiment of the technology dimension within the "Five-in-One" construction approach, and the key to solving the "high wall of usage" challenge and realizing the "ease of use" vision.

This part is divided into four chapters, respectively exploring the four technical domains of infrastructure, training support, inference support, and platform operations. As the opening chapter, Chapter 5 focuses on the starting point of all technical construction — infrastructure planning and consulting services. Infrastructure, like the foundation of a building, directly determines the height and stability that the superstructure can achieve. A poorly designed infrastructure will bring endless trouble and performance bottlenecks to subsequent platform operations and application optimization.

Therefore, we must front-load our professional technical capabilities and build a powerful infrastructure planning and consulting service system. This set of services aims to help the company's headquarters and various units "take the right path and walk steadily" at the "zero kilometer" of computing construction. We will elaborate from three levels: first, how to provide precise resource configuration consulting and optimization for specific business scenarios; second, how to plan and design data centers and computing clusters from a more macro perspective; and finally, how to establish a "computing-storage-network" integrated indicator monitoring system, providing "ECG-like" insight into the healthy operation of the infrastructure.

Computing Resource Configuration Consulting and Optimization

"How many resources do I need?" This is the first and most core question every AI application developer encounters. Too many resources is huge waste; too few, and the model cannot run or the service cannot sustain the load. The computing resource configuration consulting and optimization service is designed precisely to help users scientifically and accurately answer this question. It is not a simple "Q&A" type of customer service but a data-driven, closed-loop, expert service process spanning the entire application life cycle.

Service Channels and Forms

To allow users to conveniently access services, we have established a multi-channel service system combining online and offline methods:

Online Self-Service

• "Computing Resource Configuration Q&A Manual": We compiled a detailed, dynamically updated Q&A manual published in the knowledge base of the computing service portal. The manual covers hundreds of common questions, from "how to choose the right GPU for my YOLOv5 training task" to "how to troubleshoot I/O bottlenecks during training." This is the primary channel for user self-service learning and quick reference.
• Intelligent Q&A Robot: Embedded in the portal is an intelligent Q&A robot trained based on large model technology, capable of understanding user natural language questions and finding the most relevant answers from the Q&A manual and knowledge base, providing 7x24 hour instant response.

Online Human Service

• i6000 Ticketing System: For complex, personalized problems, users can submit an i6000 service ticket to initiate a formal consulting request. The ticketing system routes the ticket to the corresponding technical expert based on the problem type.
• Service Hotline and Expert Desk: A dedicated computing service hotline is established. For urgent problems, it can be transferred to second-line technical experts for real-time phone guidance.

Offline In-Depth Service

• Expert Clinic: Regularly (e.g., weekly) hold "Computing Optimization Expert Clinic" sessions where users can make appointments for face-to-face in-depth exchanges with architects and performance optimization experts.
• Project Embedded Consulting: For major company-level AI projects, we assign a dedicated technical advisor who deeply participates from the project inception stage, providing full-process embedded configuration consulting services.

Core Consulting Process: The Four-Step Closed-Loop Method

We have designed a standardized "Diagnosis-Analysis-Matching-Optimization" four-step closed-loop service process, ensuring each consultation provides systematic and valuable output.

Step 1: Diagnosis and Assessment of Current Computing Status — Precise "CT Scan"

This is the foundation of all consulting services. Whether planning for a new application or optimizing an old one, the first step is to conduct a comprehensive and precise "CT scan" of its current status.

• Data collection methods:
  • Proactive inventory: For new projects, we guide users through interviews and questionnaires to sort out their business objectives, technology choices, and data status.
  • Data interface integration: For existing applications, we automatically collect massive runtime performance data by integrating data interfaces with monitoring and management systems such as Prometheus, DCGM, and CMDB.
  • Automated profiling tools: We use professional tools such as NVIDIA Nsight Systems/Compute and PyTorch Profiler to deeply profile user training or inference code, capturing fine-grained performance bottlenecks.
• Core diagnostic data:
  • Computing resource configuration and status: GPU model, quantity, P2P interconnect bandwidth, memory capacity, driver and CUDA version.
  • Storage system performance: Disk type (SSD/HDD), read/write speed, IOPS, file system type (e.g., NFS, Lustre).
  • Network infrastructure status: Computing network topology (e.g., Fat-Tree), bandwidth (25/100/200 Gbps), latency, network protocol (e.g., RoCE v2).
  • Historical runtime data: GPU/memory/CPU/memory average and peak load, network bandwidth utilization, storage I/O throughput, task queuing time, etc.
• Output: "Computing Resource Status Assessment Report"
  • This report is the final result of the diagnosis, using data and charts to clearly present the resource configuration, usage status, and potential bottlenecks of the current application (or planned application environment), forming a multi-dimensional resource usage profile. As an illustrative synthetic example, the report might point out: "This training task's average GPU utilization is only 45%, while CPU utilization is consistently above 90%, and storage read IOPS has reached a bottleneck. Preliminary diagnosis indicates the data preprocessing and loading phase is the main performance bottleneck."

Step 2: Analyze Resource Usage and Business Needs

Based on the "CT scan," we need to combine the cold performance data with the vivid business context for in-depth analysis, understanding "why this is happening."

• Deconstruct business scenarios: We work with business experts and algorithm engineers to break down macro business objectives into specific AI task scenarios. For example, "drone inspection" can be decomposed into multiple sub-scenarios with different computing characteristics, such as "high-concurrency image preprocessing," "large-scale model training," "real-time video stream inference," and "offline defect report generation."
• Quantify scenario computing demands: For each sub-scenario, we quantify the core demands on "computing-storage-network" through historical data analysis, data modeling, and industry benchmarking.
  • Peak computation volume (TFLOPS): Determines how powerful GPU computing cores are needed.
  • Memory capacity and bandwidth (GB, GB/s): For large models with long text, high-resolution images, etc., this is a key bottleneck.
  • Storage read/write speed (MB/s, IOPS): For data-intensive training, this is key to whether the GPU can be "fed."
  • Network bandwidth and latency (Gbps, μs): For large-scale distributed training and low-latency inference, this is a core element.
• Draw "Demand-Status" matching map: We compare the quantified demands with the resource status diagnosed in the first step, drawing an intuitive matching map (Gap Analysis) that clearly identifies where "resource shortcomings" and "resource redundancies" are.

Step 3: Design Computing Resource Configuration Plan

This is the core value of the consulting service, providing users with a scientific, reasonable, and implementable resource configuration plan based on the analysis from the previous two steps.

• Future-oriented incremental planning: We combine the business's current status and growth expectations for the next 1-3 years (such as user growth, data growth), as well as the evolution trend of model parameters, for capacity planning.
• Hardware-software collaborative design:
  • Hardware selection and ratio: Based on the computation/memory access characteristics of the business scenario, recommend the optimal GPU model, CPU-to-GPU ratio, memory-to-memory ratio, and storage-computing network to computing network ratio. For example, for large-scale training requiring frequent communication, recommend server nodes equipped with high-bandwidth NVLink/NVSwitch; for I/O-intensive tasks, recommend configuring large-capacity local NVMe SSDs.
  • Resource pooling and isolation design: Design a reasonable resource pool partitioning strategy. For example, partition high-performance GPUs for large-scale training into one pool, and lower-end GPUs for ordinary inference or development into another pool. Use Kubernetes namespaces and Taints/Tolerations mechanisms to achieve resource isolation between different businesses.
• Dynamic resource scheduling strategy design:
  • Priority strategy: Define the preemption priorities of different businesses, ensuring that during resource scarcity, high-priority tasks (such as online inference services) can obtain resources first, even preempting low-priority tasks (such as offline analysis).
  • Scheduling affinity/anti-affinity strategy: Design strategies to ensure that all pods of a distributed training task are scheduled to the same high-speed interconnected rack (affinity), or that multiple replicas of a high-availability service are distributed to different physical fault domains (anti-affinity).
• Output: "Computing Resource Planning Plan"
  • This plan details the recommended hardware configuration list, networking scheme, resource pool partitioning strategy, scheduling strategy, etc., along with detailed reasoning and expected performance improvement effects.

Step 4: Computing Resource Configuration Optimization Service

For applications already running online, the focus of consulting services is on "optimization."

• Identify low-efficiency scenarios and resources: Through diagnosis of the current status, we proactively identify scenarios with low "input-output ratio," such as inference services with prolonged low load, training tasks with GPU utilization consistently below 20%, etc.
• Provide "shutdown, merge, transfer, downgrade" strategies:
  • Shutdown: For applications that have been abandoned or have no business value, resolutely shut them down and reclaim all their occupied resources.
  • Merge: For multiple small inference services with similar functions and low load, suggest model fusion or service merging, deploying on the same GPU, using NVIDIA MPS (Multi-Process Service) or MIG (Multi-Instance GPU) technology to achieve time-sharing multiplexing of memory and computing units, improving resource utilization.
  • Downgrade: For applications with excessive resource allocation, provide detailed downgrade plans, such as downgrading from A100 to A30 or reducing the number of replicas.
• Output: "Computing Resource Configuration Optimization Plan"
  • This plan is presented in the form of "diagnosis report + optimization suggestions + expected benefits." As an illustrative calculation example: "It is recommended to replace the GPU of Service XX from V100 to T4, which is expected to save roughly 80% of resource costs (roughly calculated by the procurement and power-consumption baseline of the two card generations), while meeting the latency requirements of the vast majority of business scenarios."

Through this closed-loop consulting and optimization service, we have successfully transformed the "tacit knowledge" of computing experts into standardized, replicable "explicit services," helping users make the most informed decisions at every stage of computing construction.

Data Center and Computing Cluster Planning and Design

If resource configuration consulting is prescribing "precise medicine" for "individual applications," then the planning and design of data centers and computing clusters is "planning and building a modern hospital" for the entire enterprise's computing infrastructure. It requires a more macro perspective, a longer-term vision, and a more cross-disciplinary knowledge system, deeply integrating IT technology with infrastructure engineering.

Core Principles of Planning and Design

When conducting planning and design, we always follow the following core principles:

• High density and green energy efficiency: AI computing clusters are typically high-power consumption, high-heat-flux density equipment. The power consumption of a single AI server can reach 10-15 kW, dozens of times that of an ordinary server. Therefore, planning and design must prioritize power supply and cooling capabilities, actively introduce advanced technologies such as liquid cooling, and pursue extreme PUE (Power Usage Effectiveness).
• High availability and scalability: The infrastructure must have high reliability, with key components (such as power supply, network) adopting redundant design. At the same time, the architecture must have good horizontal scalability, capable of convenient "online" expansion, supporting smooth business growth.
• Balance between advanced and maturity: Actively embrace the industry's latest technologies (such as 800G networking, liquid cooling, DPU), maintaining technological leadership; but also fully consider technology maturity, ecosystem completeness, and operational complexity, avoiding blindly adopting overly radical solutions.
• Standardization and modularization: Adopt standardized cabinets, servers, and network architectures, achieving "building block" style rapid deployment and expansion. Design computing, storage, and network units in a modular fashion for flexible combination and independent upgrades.

Data Center Planning and Design: Building a "Professional Track" for a "Supercar"

An AI computing cluster, this "supercar," must run on a professional "track." The planning and design of the data center is about tailoring this track for it.

• Space structure and load-bearing planning:
  • AI servers are generally taller and heavier, requiring 48U or even higher U-number cabinets, with strict calculation and reinforcement of the machine room floor load-bearing capacity (typically requiring > 1.5 tons/square meter).
  • Sufficient pipeline space needs to be reserved for accommodating dense power lines, network cables, and liquid cooling pipes.
• Power system planning:
  • High power density power supply: Use high-power PDUs (Power Distribution Units), supporting single-cabinet power supply capacity of 20kW or even 40kW+. Power supply links need to adopt 2N redundancy, ensuring that a single utility power or UPS failure does not affect equipment operation.
  • Intelligent monitoring: Conduct refined power and status monitoring for every level of circuit from the transformer to the cabinet PDU, achieving visualized power management.
• Cooling system planning: This is the most challenging aspect of AI data center planning.
  • Air cooling solutions: For medium-to-low density clusters, efficient air cooling solutions such as "cold/hot aisle containment" and "in-row air conditioners" can be used.
  • Liquid cooling solutions: For high-density clusters (e.g., single cabinet > 30kW), liquid cooling is an inevitable choice. We evaluate multiple liquid cooling solutions:
    • Cold plate liquid cooling: Precisely cools major heat sources such as GPUs and CPUs, with mature technology and relatively easy retrofitting.
    • Immersion cooling: Immerses the entire server in insulating coolant, providing the highest cooling efficiency but also the highest requirements for equipment and operations.
  • Based on factors such as construction cost, operational complexity, and PUE target, we recommend the optimal cooling solution for different scenarios.
• "Computing-Infrastructure" linkage planning and design: We emphasize the linkage planning between IT equipment and infrastructure.
  • Site survey guidance: Provide professional site survey guidance for provincial companies' data center site selection and retrofitting.
  • Construction standard output: Formulate "AI Intelligent Computing Center Construction Standards," clarifying requirements for machine room space, power, cooling, fire protection, security, etc.
  • Demand input and drawing review: The IT team (computing planners) need to clearly "input demands" to the infrastructure team, clarifying the power consumption, heat flux density, weight, and other requirements of computing equipment for the next 3-5 years. At the same time, the IT team needs to deeply participate in the review of the infrastructure team's design drawings, ensuring that infrastructure design can meet IT requirements.

Computing Cluster Planning and Design: Building a High-Performance "Computing Engine"

On top of the data center "track," we need to carefully design the "supercar" itself — the computing cluster.

• Computing node design:
  • Design different types of computing node servers based on the different requirements of training and inference. For example, training nodes emphasize multi-GPU high-speed interconnection (NVLink), while inference nodes may focus more on CPU performance and I/O capability.
  • Standardize server BOM (Bill of Materials), achieving large-scale procurement and rapid delivery.
• Storage architecture design:
  • Layered storage: Design a layered storage architecture of "high-performance hot data layer (e.g., all-flash parallel file system) + large-capacity warm data layer (e.g., HDD/SSD hybrid storage pool) + low-cost cold data archive layer (e.g., object storage or tape library)."
  • Compute-storage separation: Adopt a compute-storage separated architecture, allowing storage resources and computing resources to scale independently, providing greater flexibility.
• Network architecture design: This is key to determining large-scale cluster performance.
  • Multi-plane networking: Plan and design physically isolated computing, storage, management, and business plane networks to avoid interference between different types of traffic.
  • Computing plane (Fat-Tree architecture): For frequent All-Reduce communication in large-scale distributed training, design a non-blocking or low-convergence-ratio Fat-Tree network architecture. Use lossless Ethernet supporting RoCE v2 or InfiniBand networks to achieve low-latency, high-bandwidth inter-node communication.
  • Intelligent network operations: Introduce network telemetry technology, performing microsecond-level fine monitoring of network traffic, latency, packet loss, etc., achieving intelligent network analysis and rapid fault localization.
• Delivery and stability planning:
  • Delivery schedule planning: Formulate a detailed project delivery plan (Gantt chart), coordinating all aspects such as hardware arrival, racking, deployment, and testing to ensure on-time project delivery.
  • Delivery quality management: Formulate strict "Computing Cluster Delivery Acceptance Standards," conducting comprehensive testing and acceptance of hardware, software, performance, stability, etc.
  • Cluster stability design: At the architecture level, maximize the overall stability (MTBF) of the cluster through redundant design, fault isolation, and automated operations.

By providing end-to-end, integrated planning and design services from data center to computing cluster, we ensure that every newly built or retrofitted computing infrastructure within the company reaches industry-leading levels, truly becoming the "solid foundation" that can support the rapid development of future intelligent business.

"Computing-Storage-Network" Integrated Indicator Monitoring System

Planning and design draw the blueprint, construction delivers the physical entity, but the vitality of the infrastructure lies in healthy, stable operation. To achieve this, a comprehensive, in-depth, integrated indicator monitoring system must be established, like installing dense "sensors" and an intelligent "central control room" for the entire computing edifice. The goal of this system is to shift from passive "post-fault response" to proactive "health perception" and "problem prediction," identifying and solving problems before they occur.

Design Philosophy of the Monitoring System

• Layered decoupling, comprehensive coverage: The monitoring system needs to cover every layer from the physical layer (servers, switches), virtualization/container layer, to the system software layer, but each layer should be loosely coupled for independent upgrade and expansion.
• Unified platform, data fusion: All "computing-storage-network" monitoring data (Metrics, Logs, Traces) should be aggregated into a unified, open monitoring platform (such as Prometheus + Loki + Jaeger/Tempo), breaking data silos and enabling cross-domain correlation analysis.
• Indicator standardization: Define the company's unified core infrastructure monitoring indicator library, ensuring that equipment from different regions and manufacturers can be measured and compared using a unified standard.
• From "Availability" to "Health": The goal of monitoring is not only to judge whether equipment is "alive or dead" (availability) but also to assess "how well it is running" (health). For example, whether the network is sub-healthy (occasional high latency) or whether storage has performance jitter.

Computing Equipment Indicator Monitoring

• Hardware health monitoring:
  • Through out-of-band management interfaces (such as iDRAC/iLO), collect physical health indicators of the server, such as fan speed, CPU/memory/motherboard temperature, voltage, and power status.
  • Through tools such as NVIDIA DCGM (Data Center GPU Manager), collect GPU hardware health indicators such as temperature, power consumption, PCIe link errors, ECC errors (single-bit/double-bit), and NVLink status. Any hardware alert may indicate a potential fault.
• Resource utilization monitoring:
  • CPU/memory utilization: Collect the utilization of the entire machine and each core, system load, memory usage, and swap space usage.
  • GPU utilization: Collect the GPU computing unit utilization (GR Engine Usage), memory controller utilization, memory usage, codec engine utilization, etc.
• Output: "Computing Equipment Operations Indicator Monitoring Report"
  • This report regularly (e.g., daily/weekly) summarizes the health status and resource usage trends of computing nodes, automatically identifying "high-risk" nodes (such as GPUs with frequent ECC errors) and "low-efficiency" nodes (such as those with prolonged low load), providing input for operations and optimization.

Storage Equipment Indicator Monitoring

• Hardware health monitoring: Collect storage controller status, disk health status (SMART information), RAID group status, power/fan status, etc.
• Storage resource health monitoring:
  • Capacity indicators: Total capacity, used capacity, available capacity, Inode usage rate of the file system/storage pool. Set capacity threshold alerts to prevent space exhaustion.
  • Performance indicators: IOPS (I/O operations per second), throughput (MB/s), average read/write latency. Through continuous monitoring of these indicators, determine whether the storage system has performance bottlenecks or jitter.
  • Service status: Daemon process status of storage services such as NFS/Lustre, client mount status.
• Output: "Storage Equipment Operations Indicator Monitoring Report"
  • The report displays the capacity growth trend, performance heat map (identifying hot files or directories), latency distribution, etc., of the storage system, helping administrators with capacity planning and performance tuning.

Network Equipment Indicator Monitoring

The network is the part most prone to performance bottlenecks and the most difficult to troubleshoot in AI clusters. Refined network monitoring is crucial.

• Management plane network resource health monitoring (i.e., out-of-band management network):
  • Monitor the port status, traffic, CPU/memory utilization of management switches, ensuring smooth flow of the operations channel.
• Business plane network resource health check (i.e., computing/storage network):
  • Equipment health: Monitor the CPU/memory utilization, temperature, power/fan status of core/aggregation/access switches.
  • Port status and traffic: Monitor the Up/Down status, inbound/outbound traffic (bps), packet loss rate, and error rate of each physical port in real time. High packet loss rate or error rate is a clear signal of network faults.
  • Lossless network feature monitoring (for RoCE): Through switch telemetry, monitor key indicators such as PFC (Priority-based Flow Control) deadlock alerts and ECN (Explicit Congestion Notification) marking counts, assessing the health of the lossless network.
• End-to-end connectivity and performance real-time monitoring:
  • We deploy network performance monitoring probes (such as Pingmesh) to conduct periodic, mesh-like connectivity and latency tests between all computing nodes, drawing a global network latency topology map.
  • This can proactively discover those "sub-healthy" links that are physically connected but logically have high latency or instability, locating problems before users report them.
• Output: "Network Equipment Operations Indicator Monitoring Report"
  • The report not only provides traditional traffic and port status but, more importantly, provides a dynamic, global network quality map and can automatically alert for deep-level issues such as "abnormally increased network latency between two specific racks."

Chapter Summary

In Chapter 5, we have laid a solid foundation for the technical support system of the computing platform. Through computing resource configuration consulting and optimization services, we ensure that the resource configuration of every application is scientific and reasonable; through data center and computing cluster planning and design services, we ensure that the physical foundation of the computing infrastructure is advanced, reliable, and scalable; through the establishment of the "computing-storage-network" integrated indicator monitoring system, we have equipped this complex system with "eyes" and "ears," making its health status perceivable, measurable, and predictable. Having completed the construction of this infrastructure "ballast stone," we can confidently enter the next chapter to tackle the most core and challenging "amphibious operation" in AI applications — the technical support for training and inference.