祝融说。 GPU Dead Node Detector (GDND)

GPU Dead Node Detector (GDND)

2026.01.18

GDND is a proactive GPU health monitoring and fault isolation system for Kubernetes clusters. It runs as a DaemonSet on all GPU nodes, detecting unhealthy GPUs through multi-level inspections and automatically isolating faulty nodes via Taint/Cordon mechanisms.

Core Features

Three-tier detection pipeline
- L1 passive detection (30 seconds): NVML queries, XID error scanning, zombie process detection
- L2 active detection (5 minutes): CUDA 128x128 matrix multiplication microbenchmark
- L3 PCIe detection (24 hours, optional): PCIe bandwidth test
Health state machine: HEALTHY → SUSPECTED → UNHEALTHY → ISOLATED
Automatic isolation: Cordon node, apply Taint, evict Pod (configurable)
Prometheus metrics: Full observability support, including gdnd_gpu_status, temperature, utilization, and other metrics
Lightweight: Target image < 50MB, minimal resource footprint (10m CPU, 32Mi memory)
Extensible: Device abstraction layer supports NVIDIA GPUs and Huawei Ascend NPUs

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         GDND DaemonSet                          │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐             │
│  │ L1 passive  │  │ L2 active   │  │ L3 PCIe     │  detectors  │
│  │ (30 seconds)│  │ (5 minutes) │  │ (24 hours)  │             │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘             │
│         │                │                │                     │
│         └────────────────┼────────────────┘                     │
│                          ▼                                      │
│              ┌───────────────────────┐                          │
│              │    health state     │                          │
│              │  HEALTHY → SUSPECTED│                          │
│              │  → UNHEALTHY → ISOLATED│                         │
│              └───────────┬───────────┘                          │
│                          │                                      │
│         ┌────────────────┼────────────────┐                     │
│         ▼                ▼                ▼                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐             │
│  │   Cordon    │  │    Taint    │  │   alert     │  isolation  │
│  └─────────────┘  └─────────────┘  └─────────────┘             │
└─────────────────────────────────────────────────────────────────┘

Quick Start

Prerequisites

Kubernetes cluster 1.25+
NVIDIA GPU nodes with drivers installed
kubectl configured to access the cluster

Install with Helm (Recommended)

# Install from local chart
helm install gdnd ./release/rust/gdnd/chart \
  --namespace kube-system \
  --set config.dryRun=true  # Start in dry-run mode first for safety

# After verifying logs are clean, disable dry-run
helm upgrade gdnd ./release/rust/gdnd/chart \
  --namespace kube-system \
  --set config.dryRun=false

Install with kubectl

cd release/rust/gdnd/deploy

# Apply RBAC
kubectl apply -f rbac.yaml

# Apply ConfigMap
kubectl apply -f configmap.yaml

# Deploy DaemonSet
kubectl apply -f daemonset.yaml

Verify Installation

# Check DaemonSet status
kubectl get daemonset gdnd -n kube-system

# View logs
kubectl logs -l app.kubernetes.io/name=gdnd -n kube-system -f

# Check metrics
kubectl port-forward -n kube-system daemonset/gdnd 9100:9100
curl http://localhost:9100/metrics | grep gdnd_gpu

Configuration

Main Configuration Options

Parameter	Description	Default
`device_type`	Device type: `auto`, `nvidia`, `ascend`	`auto`
`l1_interval`	L1 passive detection interval	`30s`
`l2_interval`	L2 active detection interval	`5m`
`health.failure_threshold`	Number of consecutive failures before marking as UNHEALTHY	`3`
`health.fatal_xids`	Fatal XID error codes (immediate isolation)	`[31, 43, 48, 79]`
`health.temperature_threshold`	Temperature threshold (in Celsius)	`85`
`isolation.cordon`	Whether to cordon unhealthy nodes	`true`
`isolation.evict_pods`	Whether to evict Pods	`false`
`isolation.taint_key`	Taint key name	`nvidia.com/gpu-health`
`isolation.taint_effect`	Taint effect	`NoSchedule`
`dry_run`	Log only, no operational changes	`false`

Configuration Example: config.yaml

device_type: auto
l1_interval: 30s
l2_interval: 5m

health:
  failure_threshold: 3
  fatal_xids: [31, 43, 48, 79]
  temperature_threshold: 85
  active_check_timeout: 5s

isolation:
  cordon: true
  evict_pods: false
  taint_key: nvidia.com/gpu-health
  taint_value: failed
  taint_effect: NoSchedule

metrics:
  enabled: true
  port: 9100

dry_run: false

Fatal XID Error Codes

The following XID errors trigger immediate GPU isolation:

XID	Description
31	GPU memory page error / MMU failure
43	GPU halted processing
48	Double-bit ECC error
79	GPU detached from bus

Prometheus Metrics

Metric Name	Type	Labels	Description
`gdnd_gpu_status`	Gauge	gpu, uuid, name	Health status (0=healthy, 1=suspected, 2=unhealthy, 3=isolated)
`gdnd_gpu_temperature_celsius`	Gauge	gpu	GPU temperature
`gdnd_gpu_utilization_percent`	Gauge	gpu	GPU utilization
`gdnd_gpu_memory_used_bytes`	Gauge	gpu	GPU memory usage
`gdnd_check_duration_seconds`	Histogram	level, gpu	Detection duration
`gdnd_check_failures_total`	Counter	level, gpu, reason	Total detection failures
`gdnd_isolation_actions_total`	Counter	action	Total isolation actions
`gdnd_gpu_count`	Gauge	-	Number of detected GPUs

Development

Rust 1.75+
CUDA Toolkit 12.2+ (for compiling the gpu-check binary)

Build from Source

cd src/rust/gdnd

# Check compilation
cargo check

# Run tests
cargo test

# Build release version
cargo build --release

# Run locally (dry-run mode)
cargo run -- --config configs/config.yaml --node-name test-node --dry-run

Build Docker Image

cd release/rust/gdnd

# Build release binary
./build.sh

# Build Docker image
./build.sh --docker

Project Structure

src/rust/gdnd/
├── gdnd/                    # Main program
│   └── src/
│       ├── main.rs          # Entry point
│       ├── config.rs        # Configuration
│       └── cli.rs           # Command-line arguments
├── gdnd-core/               # Core detection logic
│   └── src/
│       ├── device/          # Device abstraction
│       │   ├── interface.rs # DeviceInterface trait
│       │   ├── nvidia.rs    # NVIDIA implementation
│       │   └── mock.rs      # Test mock
│       ├── detection/       # Detectors
│       │   ├── l1_passive.rs
│       │   └── l2_active.rs
│       ├── state_machine.rs # Health state machine
│       ├── scheduler.rs     # Detection scheduler
│       └── metrics.rs       # Prometheus metrics
├── gdnd-k8s/                # Kubernetes integration
│   └── src/
│       ├── client.rs        # K8s client
│       └── node_ops.rs      # Node operations
└── gpu-check/               # CUDA microbenchmark
    └── gpu_check.cu         # 128x128 matrix multiplication

release/rust/gdnd/
├── build.sh                 # Build script
├── chart/                   # Helm chart
├── configs/                 # Production configurations
└── deploy/                  # K8s deployment manifests

Comparison with Other Solutions

Feature	GDND	Node Problem Detector	DIY Scripts
GPU-specific detection	✅ XID, ECC, driver deadlocks	❌ Generic	Varies
Active health checks	✅ CUDA matrix multiplication	❌	Varies
Automatic isolation	✅ Cordon + Taint	⚠️ Requires manual rules	⚠️
Image size	< 50MB	~100MB	Varies
Configuration method	Simple YAML	Complex	Custom
Prometheus metrics	✅ Built-in	✅	Manual setup required

Acknowledgments

nvml-wrapper - Rust bindings for NVIDIA NVML
kube-rs - Rust Kubernetes client
prometheus-rs - Rust Prometheus client

writings