祝融说。

At this point, we have completed the entire journey from hardware selection and platform construction to model training and inference serving. The intelligent computing center we have built is like a precision, high-speed factory, performing massive calculations every second. However, complexity and fragility are twin siblings. In this factory, a tiny link -- an overheating GPU, a congested network port, a misconfigured inference engine -- can trigger a chain reaction, causing expensive training tasks to fail or online services to go down.

How can we see inside this dark factory, warn of risks in advance, and quickly locate faults? The answer is to build a full-link, multi-layered observability system.

"Monitoring" tells us "where the system went wrong," while "observability" aims to answer "why the system went wrong." It is not just about collecting metrics, but about correlating data (Metrics, Logs, Traces) from different layers -- hardware, platform, application -- to provide a drillable, analyzable global view.

In this chapter, using the industry-standard Prometheus + Grafana technology stack, we will build a three-layer monitoring system covering "hardware-platform-business" for the AIDC from scratch. We will learn how to extract the most core metrics from NVIDIA and Huawei hardware, how to extract key performance data from business applications like vLLM, and ultimately aggregate all this data into a meticulously designed "AIDC Operations Cockpit" Grafana dashboard. This dashboard will become your "control center" for managing the entire intelligent computing center.

9.1 The Collection Layer: DCGM-Exporter (NVIDIA) and NPU-Exporter (Huawei) Metric Collection

The first step of observability is data collection. For an AIDC, the most basic and important monitoring data comes from the underlying AI accelerator hardware. We need to know the health and load status of every GPU and NPU in real time.

Prometheus is a time-series database based on a pull model. It periodically accesses an HTTP endpoint (typically /metrics) exposed by a target and scrapes metric data in its specific format. To allow Prometheus to "understand" the status of GPUs and NPUs, we need an "interpreter" -- an Exporter.

9.1.1 Monitoring NVIDIA GPUs: DCGM-Exporter

NVIDIA provides a powerful suite of tools for managing data center GPUs, called DCGM (Data Center GPU Manager). DCGM is far more powerful than the commonly used nvidia-smi. It can provide richer metrics and health checks at higher frequencies and lower overhead. DCGM-Exporter is the official tool that converts the massive metrics collected by DCGM into a format Prometheus can recognize.

DCGM's Core Advantages:

• High-Performance Collection: DCGM collects data at the GPU driver level with minimal overhead.
• Rich Metrics: Beyond what nvidia-smi can show (GPU utilization, memory usage, temperature, power), DCGM also provides deeper metrics like:
  • SM Clock/Memory Clock: The actual operating frequency of the SM (Streaming Multiprocessor) and memory.
  • PCIe Replays: The number of PCIe bus retransmissions, an important indicator for diagnosing hardware link issues.
  • NVLink Bandwidth/Errors: NVLink bandwidth utilization and error counts.
  • XID Errors: Key GPU internal error codes, crucial clues for troubleshooting severe faults like "GPU falling off the bus."
  • ECC Errors: Memory error correction code counts, divided into Correctable and Uncorrectable. Uncorrectable ECC errors typically indicate GPU hardware failure.
• Proactive Health Checks: DCGM can actively diagnose GPUs, such as memory stress tests and PCIe bandwidth tests.

Deploying DCGM-Exporter (K8s DaemonSet example):

In a Kubernetes cluster, the best practice is to deploy DCGM-Exporter as a DaemonSet, ensuring an Exporter instance runs on every GPU node.

apiVersion: apps/v1
kind: DaemonSet
metadata:
    name: dcgm-exporter
    namespace: monitoring
spec:
    selector:
    matchLabels:
        app.kubernetes.io/name: dcgm-exporter
    template:
    metadata:
        labels:
        app.kubernetes.io/name: dcgm-exporter
    spec:
        nodeSelector:
        nvidia.com/gpu: "true" # Run only on nodes with NVIDIA GPUs
        containers:
        - image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.0-3.1.8-ubuntu22.04
            name: dcgm-exporter
            ports:
            - name: metrics
                containerPort: 9400
            securityContext:
            runAsUser: 0
        # ... needs to mount necessary devices and directories

Configuring Prometheus Scraping:

You need to ensure Prometheus can auto-discover these Exporters. In K8s, this is typically achieved by adding specific annotations to the Pod template of the DaemonSet. Prometheus Operator automatically generates scrape configurations based on these annotations.

# Pod template annotations
metadata:
    annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/path: '/metrics'
    prometheus.io/port: '9400'

Key DCGM Metrics Interpretation (used in PromQL):

• DCGM_FI_DEV_GPU_UTIL: GPU utilization (%), equivalent to nvidia-smi's GPU-Util.
• DCGM_FI_DEV_FB_USED: Used memory size (MB).
• DCGM_FI_DEV_POWER_USAGE: Power consumption (W).
• DCGM_FI_DEV_GPU_TEMP: GPU core temperature (C).
• DCGM_FI_DEV_XID_ERRORS: XID error count. rate(DCGM_FI_DEV_XID_ERRORS[5m]) > 0 is a very critical alert rule.
• DCGM_FI_DEV_UNCORRECTED_SBE_ERRORS / DBE_ERRORS: Uncorrectable ECC errors. An increasing trend typically means hardware needs replacement.
• DCGM_FI_PROF_NVLINK_TX_BYTES, RX_BYTES: NVLink transmit/receive bytes. Using rate() calculates real-time bandwidth.
• DCGM_FI_PROF_PCIE_RX_BYTES, TX_BYTES: PCIe transmit/receive bytes.

9.1.2 Monitoring Huawei Ascend NPUs: NPU-Exporter

For the Huawei Ascend platform, the community and Huawei provide a similar Exporter -- commonly called NPU-Exporter. Its principle is identical to DCGM-Exporter: it calls npu-smi or CANN's underlying APIs to obtain NPU status and converts it to Prometheus format.

Deploying NPU-Exporter:

Similarly deployed as a DaemonSet on all Ascend nodes. You need to obtain its container image and deployment YAML from Huawei's AscendHub or related open-source communities.

# Example DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
    name: npu-exporter
    namespace: monitoring
spec:
    template:
    spec:
        nodeSelector:
        huawei.com/npu: "true" # Run only on Ascend nodes
        containers:
        - image: ascendhub.huawei.com/public-ascendhub/npu-exporter:latest # Please refer to official source for image address
            name: npu-exporter
            ports:
            - name: metrics
                containerPort: 9101 # Port may differ
        # ...

Key NPU Metrics Interpretation:

The metric names exposed by NPU-Exporter may vary by version, but the core concepts are the same as DCGM.

• npu_utilization_ratio: NPU utilization, may include multiple dimensions like AI Core, AI CPU, and Control CPU utilization. AICore_utilization is the most important, reflecting the busyness of the core compute unit.
• npu_memory_used_bytes: Used HBM (High Bandwidth Memory) size.
• npu_temperature_celsius: NPU chip temperature.
• npu_power_watts: Power consumption.
• npu_hbm_bandwidth_usage_ratio: HBM bandwidth utilization. This is a very important performance metric; if it is low, it may indicate a "memory wall" problem.
• npu_roce_bandwidth_bytes_total: RoCE network transmit/receive bytes, used to monitor communication traffic for distributed training.

Summary: By deploying DCGM-Exporter and NPU-Exporter, we have completed the "physical layer" data collection of the observability system. Now, Prometheus is continuously receiving the "heartbeat" and "blood pressure" data from every AI accelerator card. This is the foundation for all subsequent alerting and visualization analysis.

9.2 The Business Layer: Monitoring Token Generation Rate, Request Queue Length, and Memory Fragmentation

Monitoring only hardware is far from sufficient. A service running at 100% GPU utilization might provide a terrible user experience due to massive request backlogs. We need to go deep into the "business application" and collect metrics that directly reflect service quality and efficiency. For large model inference services (e.g., those built on vLLM or TensorRT-LLM), we need to focus on several types of core business metrics.

9.2.1 How to Expose Business Metrics from an Application?

Modern inference engines (like vLLM, Triton) typically have built-in Prometheus Exporter functionality. You just need to enable an option at startup, and it will automatically expose a /metrics endpoint.

• vLLM Example: vLLM's AsyncLLMEngine and API Server have integrated Prometheus metrics. You can get rich business data directly from its /metrics endpoint.
• Triton Example: Triton Inference Server natively supports Prometheus metrics, exposed by default at http://<triton-server>:8002/metrics.
• Custom Application Implementation: If your application does not have built-in support, you can easily add custom metrics using Prometheus's official Python client library (prometheus-client).

from prometheus_client import Counter, Gauge, Histogram, start_http_server

# Define metrics
REQUESTS_IN_QUEUE = Gauge('my_app_requests_in_queue', 'Number of requests waiting in the queue')
TTFT_HISTOGRAM = Histogram('my_app_ttft_seconds', 'Time to first token histogram')

# Update metrics in your code logic
def handle_request(request):
    REQUESTS_IN_QUEUE.inc()
    # ... process request ...
    ttft = measure_ttft()
    TTFT_HISTOGRAM.observe(ttft)
    REQUESTS_IN_QUEUE.dec()

# Start an HTTP server to expose metrics
start_http_server(8000)

9.2.2 Key Business Metrics Explained

Request and Queue Metrics (Reflecting Service Load and Health)

• llm_requests_in_queue_total (Gauge): Number of requests waiting in the queue.
  • Monitoring and Alerting: This metric continuously increasing is the most direct signal that the service is about to be overloaded. Set an alert threshold, e.g., if the queue length exceeds a certain value (like 100) for 5 minutes, immediately alert for scaling.
• llm_requests_running_total (Gauge): Number of requests currently being processed on the GPU.
  • Monitoring: This value should correlate with your configured max_num_batched_tokens parameter, reflecting the GPU's concurrent processing capability.
• llm_requests_success_total / llm_requests_failed_total (Counter): Total number of successful and failed requests.
  • Monitoring and Alerting: rate(llm_requests_failed_total[5m]) calculates the failure rate. A sudden spike in failure rate is a sign of a major fault.

Performance and Throughput Metrics (Reflecting Service Efficiency)

• llm_generation_tokens_total (Counter): Total number of generated tokens.
  • Calculating TPS (Tokens Per Second): rate(llm_generation_tokens_total[5m]) gives the service's real-time token generation rate. This is the golden metric for measuring the service's total throughput.
• llm_prompt_tokens_total (Counter): Total number of processed prompt tokens.
  • Calculating Prompt Throughput: rate(llm_prompt_tokens_total[5m]).
• llm_time_to_first_token_seconds_bucket (Histogram): Histogram distribution of TTFT.
  • Calculating Percentile TTFT: histogram_quantile(0.99, sum(rate(llm_time_to_first_token_seconds_bucket[5m])) by (le)). Calculates the 99th percentile TTFT, key for evaluating service SLA (Service Level Agreement). For example, you can commit to "99% of requests have TTFT under 500ms."
• llm_time_per_output_token_seconds_bucket (Histogram): Histogram distribution of TPOT.
  • Calculating Percentile TPOT: histogram_quantile(0.99, ...). Reflects the worst-case generation speed.

Resource Management Metrics (Reflecting Engine Internal Efficiency)

• vllm_gpu_cache_usage_perc (Gauge): (vLLM specific) KV Cache utilization.
  • Monitoring: This value should consistently stay high (e.g., above 90%), proving that PagedAttention is working efficiently. If this value is low but requests are queuing, there may be other bottlenecks.
• Memory Fragmentation Rate (typically requires indirect calculation or a specific Exporter):
  • Calculation Method: (Total Memory Blocks - Free Memory Blocks - Used Memory Blocks) / Total Memory Blocks.
  • Monitoring: Ideally, this value should be close to 0. If it continues to rise, the engine's memory manager may have issues.
• vllm_scheduler_running_requests, swapped_requests, waiting_requests (Gauges): vLLM scheduler internal state, helping to deeply analyze the request processing flow.

By combining these business-layer metrics with the hardware-layer metrics from the previous section, we form a complete analysis chain. For example:

• Phenomenon: 99th percentile TTFT suddenly spikes.
• Analysis:
  • Check llm_requests_in_queue_total. If the queue length is also spiking -> caused by request backlog.
  • Check DCGM_FI_DEV_GPU_UTIL. If GPU utilization is already at 100% -> compute bottleneck, needs scaling.
  • If GPU utilization is not high but the queue is backlogged -> could be CPU bottleneck (Python code, API server I/O), network I/O bottleneck, or a problem with the inference engine's scheduling logic.

9.3 Visualization in Practice: Building an "AIDC Operations Cockpit" Grafana Dashboard from Scratch

Data collection is complete. Now we need to turn these cold numbers into intuitive, understandable, interactive charts. Grafana is the undisputed choice for this task. A well-designed Grafana Dashboard is not just a "combat command center" for operations personnel, but also a "Business Intelligence (BI)" panel for demonstrating operational results and reporting resource utilization to management.

We will design an "AIDC Operations Cockpit" consisting of three core parts: Global Overview, Training Cluster Monitoring, and Inference Service Monitoring.

Prerequisite: You have already deployed Prometheus and Grafana, and Prometheus is configured to scrape metrics from DCGM-Exporter, NPU-Exporter, and the inference service business metrics.

Step 1: Design Dashboard Structure (Layout & Variables)

• Create a new Dashboard.
• Use Template Variables: This is key to making the Dashboard "come alive."
  • $node: Create a variable of type Query. Query expression: label_values(node_uname_info, nodename). Used for switching between nodes.
  • $gpu: Create a variable of type Query. Query expression: label_values({__name__=~"DCGM_FI_DEV_GPU_UTIL"}, gpu). Used for switching between individual GPUs.
  • $service: Create a query to get all inference service names: label_values(llm_requests_in_queue_total, service_name).

Step 2: Build "Global Overview" (The Big Picture)

This section is for managers and the "first glance" of frontline operations, providing the most core macro indicators.

• Stat Panel: Core KPIs

  • Total GPUs/NPUs: count(count by (instance)(DCGM_FI_DEV_GPU_UTIL))
  • Average GPU Utilization: avg(DCGM_FI_DEV_GPU_UTIL)
  • Total GPU Power: sum(DCGM_FI_DEV_POWER_USAGE) / 1000 (in KW)
  • Number of GPUs in Alert: count(ALERTS{alertstate="firing", alertname=~"GPU.*"})

• Time Series Panel: Global Resource Utilization Trend

  • Query A (GPU Util): avg(DCGM_FI_DEV_GPU_UTIL) by (job) (if using K8s, can aggregate by namespace or pod)
  • Query B (Memory Util): avg(DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) * 100

• Table Panel: Node Resource Ranking

  • Sort nodes by average GPU utilization, quickly identifying high-load or low-load nodes.
  • Query: avg by (nodename) (DCGM_FI_DEV_GPU_UTIL)

Step 3: Build "Training Cluster Drill-Down View"

This section focuses on diagnosing and analyzing training tasks.

• Row: Drill-down by node and GPU

  • Use our created $node and $gpu variables.

• Time Series Panel: Single GPU Core Metrics (Repeat this panel to show all GPUs on $node)

  • GPU/NPU Util: DCGM_FI_DEV_GPU_UTIL{nodename="$node", gpu="$gpu"}
  • Memory Used: DCGM_FI_DEV_FB_USED{nodename="$node", gpu="$gpu"}
  • Power & Temp: DCGM_FI_DEV_POWER_USAGE{...}, DCGM_FI_DEV_GPU_TEMP{...}

• Time Series Panel: Network Communication Monitoring

  • NVLink Bandwidth: rate(DCGM_FI_PROF_NVLINK_TX_BYTES{...}[5m]) / 1024 / 1024 (MB/s)
  • RoCE Bandwidth: rate(npu_roce_bandwidth_bytes_total{...}[5m])
  • Significance: During distributed training, these graphs should show regular, high-peak communication traffic. If traffic is very low or absent, distributed communication may not be working properly.

• Stat Panel / Table: Hardware Error Monitoring

  • XID Errors: sum(rate(DCGM_FI_DEV_XID_ERRORS{nodename="$node"}[10m])) by (gpu)
  • ECC Errors: sum(rate(DCGM_FI_DEV_UNCORRECTED_SBE_ERRORS{...}[10m])) by (gpu)
  • Key: Any non-zero value warrants high alert!

Step 4: Build "Inference Service Drill-Down View"

This section focuses on evaluating the performance and health of online services.

• Row: Drill-down by service and instance

  • Use $service variable.

• Time Series Panel: Quality of Service (QoS)

  • 99th TTFT: histogram_quantile(0.99, sum(rate(llm_time_to_first_token_seconds_bucket{service_name="$service"}[5m])) by (le))
  • Avg TTFT: sum(rate(llm_time_to_first_token_seconds_sum[5m])) / sum(rate(llm_time_to_first_token_seconds_count[5m]))
  • Error Rate: sum(rate(llm_requests_failed_total{service_name="$service"}[5m])) / sum(rate(llm_requests_total{service_name="$service"}[5m]))

• Time Series Panel: Throughput and Load

  • TPS (Tokens/sec): sum(rate(llm_generation_tokens_total{service_name="$service"}[5m]))
  • RPS (Requests/sec): sum(rate(llm_requests_total{service_name="$service"}[5m]))
  • Queue Length: llm_requests_in_queue_total{service_name="$service"}

• Time Series Panel: Engine Internal State

  • KV Cache Usage: vllm_gpu_cache_usage_perc{service_name="$service"}
  • Running vs Waiting Requests: vllm_scheduler_running_requests, vllm_scheduler_waiting_requests

Step 5: Configure Alerting

Grafana integrates powerful alerting capabilities. You can configure alert rules for almost any panel.

Key Alert Rule Examples:

• Hardware Alerts:
  • GPU_Too_Hot: DCGM_FI_DEV_GPU_TEMP > 85 for 5m
  • GPU_XID_Error: rate(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
  • GPU_Uncorrectable_ECC: increase(DCGM_FI_DEV_UNCORRECTED_SBE_ERRORS[10m]) > 0
• Business Alerts:
  • Inference_High_TTFT: 99th TTFT > 1s for 5m
  • Inference_High_Queue_Length: llm_requests_in_queue_total > 100 for 10m
  • Inference_High_Error_Rate: Error Rate > 5% for 1m

Configure these alert rules and connect them to your notification channels (e.g., Slack, PagerDuty, DingTalk), and you have a 7x24 "intelligent sentry."

Summary:

A well-designed Grafana Dashboard is far more than a collection of charts. It is a storyboard that tells the complete story of your AIDC, from hardware to business. It is a diagnostic tool that helps you quickly drill down from macro phenomena to micro root causes. It is also a value amplifier, presenting the heavy and complex operational work you and your team do behind the scenes to stakeholders in an intuitive way. This is the appeal of observability.