祝融说。

In Chapter 9, we installed a powerful "all-seeing eye" for the intelligent computing center -- a full-link observability platform. We can now see every heartbeat of the hardware and trace every request of the business. However, monitoring alone does not solve problems; it merely sounds the battle horn. When the alert rings, the real test begins.

For a cluster of thousands of cards with an investment on the order of hundreds of millions of yuan, every minute of downtime can mean thousands or even tens of thousands of yuan in compute cost going up in smoke (an order-of-magnitude estimate depending on cluster cost and utilization). As the guardians of the AIDC, our core value lies not only in ensuring the system "does not break," but in restoring service as quickly as possible when it does -- this is MTTR (Mean Time To Recovery). This demands not only familiarity with tools, but also clinical diagnostic thinking like an emergency physician: rapid triage, locate the lesion, prescribe the right remedy, and conduct post-mortems.

In this chapter, we will confront the most challenging and thrilling aspect of AIDC operations -- fault handling. We will start from the lowest-level hardware faults (such as GPU falling off the bus and ECC errors), move upward to the headache-inducing network communication issues in distributed training (such as NCCL Timeout), and finally delve into training process problems closely tied to algorithms (such as Loss NaN and gradient explosion). What we will provide is not a scattered list of "folk remedies," but a structured, followable SRE troubleshooting decision tree that helps you maintain a clear mind in the chaos of a fault scene, step by step, straight to the root cause.

10.1 Hardware-Level Faults: ECC Errors, XID Errors, GPU Fall-off, Optical Module Faults

Hardware is the physical foundation of all computation and the layer where faults occur most frequently. Hardware faults are typically sudden, unpredictable, and severe in consequence. A mature AI Infra team must have high sensitivity to hardware faults and standardized handling procedures (SOPs).

10.1.1 ECC Errors: The Memory "Health Alert"

What is an ECC Error?

ECC (Error-Correcting Code) is a memory fault tolerance technology. Data center-grade GPUs (such as A100, H800, and 910B) with HBM memory all support ECC. It can detect and correct small-scale bit flips during data read/write operations.

Bit flips can be caused by various factors: cosmic rays, power supply noise, chip aging, and more.

ECC errors fall into two categories:

1. Correctable Errors: Single-bit errors. The ECC circuitry can automatically detect and correct them, completely transparent to the upper-layer application, and they typically do not cause program crashes.
2. Uncorrectable Errors: Two or more bit errors that exceed the ECC's correction capability. These cause data corruption and typically trigger severe GPU driver errors, leading to application crashes or GPU resets. Uncorrectable errors are classified as SBE (Single-Bit Error, at certain special locations) and DBE (Double-Bit Error).

How to Detect?

• Monitoring and Alerting (Best Practice): In the previous chapter, we already set up alert rules.
  • increase(DCGM_FI_DEV_UNCORRECTED_SBE_ERRORS[1h]) > 0 or increase(DCGM_FI_DEV_UNCORRECTED_DBE_ERRORS[1h]) > 0 -> High priority alert!
  • increase(DCGM_FI_DEV_CORRECTED_SBE_ERRORS[1h]) > 100 -> Medium priority alert.
• Manual Check:
  • NVIDIA: nvidia-smi -q -d ECC
  • Huawei Ascend: npu-smi info -t ecc

Troubleshooting and Handling Decision Tree:

if (Uncorrectable errors appear):

This is a clear hardware failure signal!

1. Immediately isolate the node: Set the node with the faulty GPU to unschedulable (kubectl cordon <node-name>) and drain all running Pods, preventing new tasks from being scheduled onto the faulty hardware.
2. Record information: Note the faulty GPU card UUID, server serial number, fault time, and error logs (dmesg | grep -i nve or similar NPU logs).
3. Create a repair ticket: Submit a ticket to the data center field team or hardware vendor requesting GPU card replacement. Do not attempt to "fix" it by rebooting -- this is physical damage and the problem will recur.
4. Post-mortem analysis: Track the failure rate of this batch of GPU cards. If it is generally high, a batch quality issue may need to be discussed with the vendor.

else if (A small number of occasional correctable errors appear):

Usually no immediate action is needed, but monitor closely.

1. Continue monitoring: Observe the growth rate of Correctable Errors on this card. If only a few appear over hours or even days, it can be temporarily ignored.
2. Check environmental factors: Verify whether the server room temperature and humidity are within normal ranges. Excessive temperature increases the probability of bit flips.

else if (Correctable errors grow continuously and rapidly):

This is a precursor to the hardware reaching its "end of life."

1. Assess risk: Although the application has not crashed yet, frequent ECC corrections incur a slight performance overhead, and this is a strong signal that it will develop into uncorrectable errors.
2. Planned maintenance: Isolate the node and schedule hardware replacement during off-peak hours. This is "preventive maintenance" to avoid a sudden failure at a critical moment.

10.1.2 XID Errors: The GPU Driver's "Distress Signal"

What is an XID Error?

• XID is a set of error codes defined inside the NVIDIA driver (historically derived from the X server's error numbering system, with no direct relation to Xorg today). When the GPU encounters a serious internal error that the driver cannot handle on its own, it prints a NVRM: Xid (PCI: ...): ... error message in the kernel log (dmesg).
• Each XID code corresponds to a specific type of internal fault, making it a "treasure map" for troubleshooting elusive GPU issues.

How to Detect?

• Monitoring and Alerting: rate(DCGM_FI_DEV_XID_ERRORS[5m]) > 0 -> High priority alert!
• Manual Check: dmesg | grep -i xid

Common XID Codes and Troubleshooting Guidance:

• XID 13 / XID 31: GPU Page Fault.

  • Meaning: The GPU attempted to access an invalid, non-existent, or unauthorized memory address.
  • Possible Causes:
    1. CUDA Program Bug (most common): Your AI application code has memory access violations, uses dangling pointers, etc. This requires algorithm engineers to debug the code.
    2. Driver Bug: In rare cases, it may be a bug in the NVIDIA driver itself. Try upgrading or downgrading the driver version.

• XID 43 / XID 45: GPU has fallen off the bus.

  • Meaning: The GPU's PCIe connection to the motherboard has been lost.
  • Possible Causes:
    1. Poor Hardware Contact: The GPU card is not fully seated, or the PCIe slot gold fingers are oxidized.
    2. Power Supply Issue: The GPU's external power cables are loose or the power supply is insufficient.
    3. Motherboard or GPU hardware failure.
  • Handling: This is a serious hardware problem. A field engineer needs to power down the server, reseat the GPU card and power cables. If the problem recurs, replace the GPU card or motherboard.

• XID 62 / XID 79: Illegal instruction or memory operation.

  • Meaning: The GPU's compute unit encountered an unrecognizable instruction or illegal operation during execution.
  • Possible Causes:
    1. Hardware Fault: A compute unit or scheduler inside the GPU is damaged.
    2. Overheating: The temperature is too high, causing the GPU to work unstably.
  • Handling: First check the GPU temperature. If the temperature is normal, this typically also points to a hardware fault requiring replacement.

• General XID Troubleshooting Flow:

  1. Record the XID code and error message.
  2. Search Google and NVIDIA Developer Forums: Enter the full XID error message into a search engine. You can often find NVIDIA's official explanation and common causes from other developers.
  3. Correlate with context: Check what task was running on the GPU when the fault occurred. Was it a specific training script? If so, ask the algorithm engineer to review the code.
  4. Hardware exclusion: If multiple different, mature applications trigger XID errors on the same card, it is likely a hardware problem. Conversely, if a certain XID only appears when running a newly developed program, it is likely a software bug.

10.1.3 GPU/NPU Falling Off the Bus: The Most Direct "Loss of Contact"

Phenomenon:

• The number of devices listed by nvidia-smi or npu-smi is fewer than the physically installed count.
• Training tasks report CUDA_ERROR_NO_DEVICE or similar NPU device not found errors.
• dmesg may contain logs such as "GPU has fallen off the bus."

Troubleshooting Decision Tree:

1. Step 1: Software-level reset.

   • NVIDIA: Try executing sudo nvidia-smi --gpu-reset -i <gpu_id>.
   • Huawei Ascend: Try using the npu-smi reset related commands.
   • if (Reset succeeds, card becomes visible again):

     This could be caused by a temporary software state anomaly or a transient driver issue. The card can be returned to service, but needs close monitoring. If it falls off again shortly, the problem is likely at the hardware level.

   • else (Reset fails):

     Proceed to the next step.

2. Step 2: Operating system level check.

   • Execute lspci | grep -i 'NVIDIA\|Huawei'.
   • if (Device not visible in lspci):

     The GPU has "disappeared" at the PCIe bus level. This is a clear hardware or firmware issue. Skip directly to Step 4.

   • else (Device visible, but driver fails to load):

     The problem may be with the driver and kernel module. Try purging and reinstalling the driver (sudo apt-get purge nvidia-* then reinstall). If the problem persists, proceed to the next step.

3. Step 3: Cold server restart.

   • Perform a complete cold start on the faulty server (shut down and power on, not reboot). A cold start reinitializes all hardware.
   • if (Card recovers after restart):

     The problem may be complex, possibly a compatibility or firmware issue between the motherboard, BIOS/UEFI, and the GPU. Record the server model, BIOS version, and GPU model, and observe whether it recurs on similar machines. Consider upgrading the server firmware.

   • else (Card still missing after restart):

     If the error reproduces consistently on the same physical node across tasks and containers, prioritize the node's hardware, firmware, or links. Do not assign a "99%" fault probability without a sample baseline and elimination steps.

4. Step 4: On-site hardware troubleshooting.

   • Isolate the node and submit a ticket.
   • Field engineer performs standard operations: a. Power down, reseat the GPU card. b. Check and reseat the GPU's external power cables. c. Swap the faulty card with a known-good card on the same server (exchange PCIe slots). If the problem follows the card (i.e., it is still faulty in a different slot), it is a GPU card failure. If the problem stays with the slot (i.e., the good card also fails in this slot), it is a motherboard PCIe slot failure. d. Replace hardware: Based on the above determination, replace the GPU card or motherboard.

10.1.4 Optical Module and Cable Faults

For large-scale distributed training that relies on RDMA networks, any imperfection in the physical network is dramatically amplified. Optical modules and cables are seemingly insignificant but extremely common points of failure.

Phenomenon:

• A node's training speed suddenly becomes extremely slow, dragging down the entire cluster.
• Training logs show a large number of NCCL Timeout or similar communication timeout errors, always pointing to the same node.
• Switch logs show a large number of CRC errors, packet drops, or frequent UP/DOWN transitions on a certain port.

Troubleshooting Approach:

1. Locate the problem port:

   • Log in to the node reported in the NCCL logs.
   • Execute ibstat (InfiniBand) or ethtool <interface_name> (RoCE) to check the network card port status. State should be Active, and Physical state should be LinkUp.
   • Log in to the switch connected to that port, and check the port's status and error counters: show interface <interface_id> counters errors.

2. Use hardware diagnostic tools:

   • Mellanox Tools (IB/RoCE):
     • ibdiagnet: A powerful IB network diagnostic tool that scans the entire network's topology, link quality, and configuration, generating detailed reports.
     • ibclearerrors: Clears error counters on all ports.
     • ibqueryerrors -c: After a period of time, query error counters again to see which ports have new error growth.
   • Switch Diagnostic Commands: Modern switches typically support reading the optical module's Digital Diagnostic Monitoring (DDM/DOM) information via commands such as show interface transceiver detail. This includes:
     • Optical module receive/transmit power (Rx/Tx Power). If the optical power is too low or too high, it indicates a problem with the optical module or fiber.
     • Temperature and voltage.

3. Physical replacement method:

   • Replace cable: Replace the fiber optic patch cable on the faulty port.
   • Replace optical module: If replacing the cable does not work, replace the optical module on the port.
   • Replace port: Plug the cable into another free port on the switch or network card.
   • Through these cross-validations, the faulty component -- cable, optical module, network card port, or switch port -- can be quickly identified.

SRE Practice Summary:

For hardware faults, the key is standardization and process orientation. You need to establish a complete closed loop covering "monitoring and alerting -> automatic isolation -> ticketing system -> field operations -> fault post-mortem." At the same time, spare parts management is critical. The data center must maintain a sufficient stock of GPUs, network cards, optical modules, cables, and other spare parts to shorten repair waiting times.

10.2 Network-Level Faults: NCCL Timeout and Communication Deadlock Troubleshooting

The network is the lifeline of distributed training. Network faults are typically more insidious and harder to reproduce than hardware faults, making them one of the "nightmares" for AI Infra engineers.

10.2.1 NCCL Timeout: The Most Common "Network Cold"

Phenomenon:

After a long period of silence, the training log suddenly prints a large number of error messages similar to the following: NCCL WARN Cuda Error in ... : 700 (an illegal memory access was encountered) NCCL WARN unhandled cuda error ... ret=700 torch.distributed.DistBackendError: NCCL error in ...: 7 (Internal check failed) The key terms are Timeout, unhandled cuda error, and Connect recv failed.

Root Cause:

The essence of NCCL Timeout is this: one or more GPUs, when participating in collective communication such as All-Reduce, did not receive data or signals from other GPUs within the specified time. This is like a relay race where one runner never passes the baton, causing the entire team to be stuck.

Troubleshooting Decision Tree:

Level 1: Is it a "fake" network problem?

Often, the network is the "scapegoat"; the true cause lies elsewhere.

1. Check if a GPU is hung: This is the most common cause. A GPU, due to hardware failure (e.g., an XID error), overheating, or a program bug, becomes unresponsive and cannot participate in communication.
   • Troubleshooting Method: Simultaneously execute nvidia-smi or check dcgm-exporter metrics on all nodes. If a GPU's utilization, power consumption, or temperature is abnormal (e.g., utilization is 0 but memory usage is high, or the temperature is sustained above 95 degrees Celsius), or if dmesg contains XID logs, the problem is on that card. Handle it per Section 10.1.
2. Check for CPU bottleneck or OOM Killer:
   • Troubleshooting Method: During training, monitor CPU usage on all nodes using top or htop. If the data preprocessing logic is too complex, causing a node's CPU to be 100% saturated, that node may not be able to submit compute or communication tasks to the GPU in time, triggering NCCL Timeout on other nodes.
   • Use dmesg | grep -i 'out of memory' to check if any process was killed by the system's OOM Killer. If a data loading process is killed, the GPU will also hang due to "starvation."

Level 2: Confirmed as a network problem, begin investigation.

If all GPUs and CPUs appear normal, the problem is likely at the network layer.

1. Enable NCCL debug logging: This is the "artifact" for troubleshooting NCCL problems. Before starting training, set the environment variables: export NCCL_DEBUG=INFO export NCCL_DEBUG_SUBSYS=ALL Rerun the task. NCCL will print extremely detailed logs, including:
   • The network topology it detected (Ring/Tree).
   • Which communication protocol it selected (LL, LL128, Simple).
   • The process of establishing connections between each pair of GPUs.
   • The detailed process of data transfers. Carefully reading these logs usually reveals between which two nodes the connection establishment failed, or at which step it got stuck.
2. Basic network connectivity test:
   • Perform basic ping and traceroute between the two nodes reporting timeout, ensuring the network is reachable.
   • RDMA connectivity test:
     • IB network: ibping
     • RoCE network: rping
     • If RDMA ping fails, it indicates an issue with the RDMA configuration (such as RoCE's PFC/ECN) or the physical link.
3. Bandwidth and latency test:
   • Use tools such as ib_write_bw, ib_read_bw (IB) or qperf (RoCE) to perform point-to-point bandwidth and latency stress tests between the faulty node pair.
   • if (Bandwidth is far below the theoretical value or latency is extremely high):

     The problem lies on the physical link between these two nodes. Follow the method in Section 10.1.4 to check the optical modules, cables, and switch ports.

4. Check lossless network configuration (RoCE specific):
   • RoCE networks fear packet loss the most. Packet loss causes RDMA performance to plummet, triggering timeouts.
   • Log in to all switches along the path and check PFC and ECN related counters.
   • show priority-flow-control counters interface ...: Check PFC PAUSE frame transmit/receive statistics. If a port has a huge number of Tx Pause frames, it indicates severe downstream congestion.
   • show queue-counters interface ...: Check queue drop counts. Any non-zero packet drops indicate a failed lossless network configuration.
   • Debugging lossless networks is very complex, requiring network expert intervention to check end-to-end DSCP priority markings, switch queue mappings, WRED/ECN thresholds, and other configurations.

10.2.2 Communication Deadlock

Phenomenon:

The training task appears to be running; all GPU utilization may be high (even 100%), but the training step or loss does not update for a long time. The task is "alive" but "dead."

Cause:

Deadlocks typically occur in more complex parallelism strategies (such as tensor parallelism combined with pipeline parallelism) or custom communication logic. Their essence is the formation of a circular waiting dependency.

• Example:
  • GPU 0 is waiting for data from GPU 1.
  • GPU 1 is waiting for data from GPU 2.
  • ...
  • GPU N is waiting for data from GPU 0.
• This cycle prevents any GPU from making progress.

Troubleshooting Approach:

1. Attach with GDB/pdb: This is the most direct but also the most hardcore method. Log in to one of the stuck nodes, find the training Python process, and use tools like gdb -p <pid> or py-spy to attach to it and inspect the stack of all threads.
   • py-spy top --pid <pid>: Shows the time spent in each function in real time. If all time is consumed in a communication call such as torch.distributed.recv or hccl.recv, deadlock is confirmed.
   • In gdb, use the bt command to print the stack. If the top of the stack is a blocking communication operation, that also points to the problem.
2. Simplify parallelism strategy:
   • If using complex 3D parallelism, try reducing the dimensions. For example, first remove pipeline parallelism and run with just TP+DP to see if it still deadlocks. Then remove tensor parallelism and run with just DP. Through this approach, isolate which parallelism strategy implementation or interaction is causing the problem.
3. Code review:
   • The root cause of a deadlock always lies in the code logic. Algorithm engineers and framework engineers need to carefully review the communication order in the model's forward function.
   • Ensure there is no circular dependency.
   • Ensure the order of communication operations (such as send/recv) is consistent across all ranks.
   • Check for cross-communication between different process groups, which is especially prone to causing deadlocks.

10.3 Training-Level Faults: Loss NaN, Gradient Explosion, and Training Stuck -- Operations Troubleshooting Tree

These types of faults occur at the algorithmic level of the training process, but their root causes can come from data, code, hyperparameters, or even hardware. While AI Infra engineers are not directly responsible for modifying algorithms, they need to provide the necessary tools and troubleshooting guidance to help algorithm engineers quickly locate problems.

10.3.1 Loss NaN (Not a Number): Training "Derailed"

• Phenomenon: The loss value in the training log suddenly becomes NaN. Once it appears, it is usually irreversible, and the training run is effectively a failure.

• Root Cause: A mathematically undefined operation occurred during computation, such as 0/0, sqrt(-1), or log(0).

• Troubleshooting Decision Tree (Operations Perspective):

Step 1: Check the data.

"Garbage in, garbage out." Dirty data is a common culprit for NaN.

1. Data loading script: Check the data preprocessing and loading code for logic errors that might result in reading empty files, damaged images, or generating empty token sequences.
2. Numerical stability: Check whether the input data has been normalized. If the input numerical range is too large or too small, it can easily overflow during computation.

Step 2: Check the model and hyperparameters.

1. Learning rate too high: This is the most common cause! An excessively high learning rate causes parameter updates to take steps that are too large, directly "jumping" into a region that produces NaN.
   • Operations suggestion: Recommend that the algorithm engineer reduce the learning rate (e.g., by an order of magnitude) and restart training from the last good checkpoint.
2. Numerically unstable operations in the model implementation:
   • For example, when computing cross-entropy loss, if exp() and then log() are applied to logits, when logits are very large, exp() may overflow to inf, and log(inf) is still inf, but the intermediate division might produce inf/inf, resulting in NaN. PyTorch's torch.nn.CrossEntropyLoss internally handles these numerical stability issues.
   • Operations suggestion: Recommend that the algorithm engineer check for unsafe custom mathematical operations and prefer using framework-provided, numerically stable modules.
3. Mixed precision training issues:
   • When using FP16 half-precision training, its numerical range (approximately 6e-5 to 65504) is much smaller than FP32. If gradients are too small, they may underflow to 0; if values are too large, they may overflow to inf.
   • Loss Scaling: Tools like PyTorch's AMP (torch.cuda.amp.GradScaler) automatically perform loss scaling to mitigate this problem. After computing the loss, it multiplies it by a large scaling factor (e.g., 65536), correspondingly amplifying the gradients to avoid underflow. Before updating parameters, it divides the gradients by the scaling factor.
   • Operations suggestion: Ensure the algorithm engineer has correctly enabled mixed precision training and Loss Scaling. If NaN still occurs, try adjusting the GradScaler's init_scale parameter.

Step 3: Use debugging tools.

PyTorch Anomaly Detection: Add these two lines at the beginning of the training script:

import torch
torch.autograd.set_detect_anomaly(True)

When NaN occurs, PyTorch will print the complete backpropagation stack trace that led to the NaN, directly telling you which operation produced the bad gradient. This is the ultimate weapon for locating the problem. Note that it slows down training, so only enable it during debugging.

10.3.2 Gradient Explosion and Vanishing

• Phenomenon:
  • Gradient Explosion: The loss value suddenly increases dramatically, becoming a huge number or even inf.
  • Gradient Vanishing: The loss stops decreasing for a long time, or decreases extremely slowly. The model is unable to learn anything.
• Troubleshooting Approach:
  • Gradient Clipping: This is the standard method for dealing with gradient explosion. Before the optimizer updates parameters, it checks the gradient's norm. If it exceeds a threshold, it is "pulled back" within the threshold.
    • Operations suggestion: Confirm with the algorithm engineer whether torch.nn.utils.clip_grad_norm_ has been added to the training code.
  • Weight Initialization: Improper weight initialization methods are one of the root causes of gradient vanishing and explosion.
  • Check Model Architecture: Using inappropriate activation functions (such as Sigmoid, which easily leads to gradient vanishing in deep networks), or not using Batch Normalization or Layer Normalization, etc.

10.3.3 Training Stuck

This is a manifestation of "communication deadlock" (Section 10.2.2) at the application level, but the causes can be broader.

Troubleshooting Approach:

1. First eliminate network deadlock: Follow the method in Section 10.2.2 to check the communication stacks.
2. Check data loading: Is the data loading process (DataLoader workers) stuck?
   • Cause: Could be slow disk I/O, or a worker that crashed due to a bug.
   • Troubleshooting: Log in to the node, run ps aux | grep python, and check for zombie processes. Monitor disk I/O (iostat).
3. Infinite loop bug: Does the training code itself contain an infinite loop?
   • Troubleshooting: Use py-spy to attach to the process and check which function is running endlessly.
4. Resource contention: Are there other high-priority processes (even monitoring agents deployed by operations) preempting CPU or memory resources, causing the training process to "starve"?

SRE Practice Summary:

For training-level faults, the AI Infra team's role is that of "enabler" and "tool provider." You need to:

1. Provide a debugging environment: Allow algorithm engineers to easily enter a problematic Pod for interactive debugging.
2. Provide debugging tools: Pre-install debugging tools such as py-spy and gdb in the base image.
3. Provide monitoring data: Incorporate algorithm-level metrics such as gradient norm, learning rate, and loss curves into monitoring, correlating them with hardware metrics.
4. Build a Knowledge Base: Document the cause, troubleshooting process, and solution for each fault, forming the team's "error handbook."

Through the study of this chapter, you have not only mastered how to handle various typical faults spanning hardware to software, but more importantly, you have established a structured SRE troubleshooting mindset. This thinking ability is far more valuable than memorizing any specific command. It will be your most reliable asset as a top-tier AI Infra engineer when facing the endless stream of new problems in the future.