祝融说。

In the previous chapters, we have built for the GPU cluster a powerful computing core (GPU servers), a high-speed internal "neural network" (the compute fabric), and a flexible virtual "urban plan" (network virtualization). Now we arrive at the final, equally indispensable cornerstone of this "AI city" — storage.

The storage system is at once the "granary" and the "library" of the entire GPU cluster. It hosts everything: operating systems, applications, massive training datasets, intermediate checkpoints, and the models that are finally produced. If the storage system underperforms and data supply cannot keep up, then no matter how powerful the GPUs or how fast the network, the whole cluster can only gaze helplessly from afar, and its efficiency will plummet. As an industry adage puts it: "In high-performance computing, for every dollar you spend on compute, you must spend the same on I/O to support it."

And yet designing storage for a large-scale GPU cluster is far more than picking a "big-enough hard drive." AI workloads place demands on storage that are diverse, multi-layered, and exacting.

Operating systems and programs require a stable, reliable, low-latency storage volume.

Raw datasets on the order of petabytes (images, video, text) demand a storage pool that is enormous in capacity, cost-controllable, and easy to manage.

During distributed training, hundreds or thousands of compute nodes need to read the same training data concurrently and at high performance, posing an extreme challenge to the storage system's concurrent read throughput.

The massive model checkpoints produced during training must be written quickly, to minimize training interruption time.

No single storage technology can perfectly satisfy all of these requirements. A successful GPU cluster storage architecture must therefore be a layered, heterogeneous combination that deploys different technologies against different needs.

In this chapter, acting as storage architects, we will systematically explore how to design and implement a high-performance, highly available layered storage system for a GPU cluster. Organizing our discussion around the three typical storage requirements that arise in an AI workflow, we will dissect three mainstream distributed storage technologies:

1. Program and system storage — distributed block storage: We will examine why distributed block storage is needed and how it furnishes virtual machines and containers with high-availability "cloud disks" that behave like local drives. We will use Ceph RBD, the most widely deployed open-source project in this space, as our example and delve into its principles.
2. Massive unstructured data storage — distributed object storage: We will see how object storage, with its near-limitless scalability, extremely low cost, and simple HTTP interface, becomes the ideal choice for housing enormous raw datasets. We will introduce its leading representatives, Ceph RGW and MinIO.
3. AI training data storage — distributed concurrent high-performance storage: This is the focus of the chapter. We will concentrate on solving the "data feeding" problem in large-scale parallel training, taking a deep look at the distributed parallel file systems born for HPC and AI — the classics Lustre and GPFS — and at how they combine with technologies such as GPUDirect Storage.

By the end of this chapter, you will have mastered a complete methodology for designing GPU cluster storage architecture: you will be able to select the most appropriate storage technology for each data type and access pattern, and to compose them into a powerful data foundation that can both "hold it all" and "serve it fast," supplying a steady stream of "data fuel" to your AI cluster.

10.1 Program and System Storage — Distributed Block Storage

In our GPU cluster, whether the operating system runs on a virtual machine or a bare-metal server, and whether the application is deployed in a container, everything needs a place to be installed and to run. That "place" is its root filesystem, which must be mounted on a block device.

In a traditional single-machine environment, this block device is a local SATA or NVMe SSD. But in a large-scale, cloudified cluster, relying on local disks brings numerous problems: poor reliability (a failing local disk can lose an entire system or application), no ability to migrate, and cumbersome management. We therefore need a networked, distributed block storage that can hand any compute node a highly available, well-performing virtual disk, much like a "plug-and-play" USB drive. This is the origin of the "cloud disk" service, and its underlying technical core is precisely distributed block storage.

10.1.1 Business Requirements for Block Storage

1. A stable, reliable system disk: It provides the boot disk for the operating systems of hundreds or thousands of compute nodes. These disks must be highly available — even if a backend physical disk or a storage server fails, the operating system's operation should not be interrupted.
2. Persistent container volumes: Containers are stateless by nature, but many applications (such as databases and middleware) need to persist data. Distributed block storage can give such stateful containers a persistent volume (PV) whose lifecycle is independent of the container. When a container migrates between nodes, this volume can be easily "unmounted" and "remounted."
3. Virtual machine disks: In a virtualized environment, each VM's virtual disk (such as VMDK or qcow2 files) needs to reside on shared, high-performance storage so as to support advanced features like VM creation, snapshots, and migration. Distributed block storage is the ideal backend for hosting these virtual disk files.
4. Low latency and moderate throughput: The access pattern of system and program disks is typically a high volume of random small reads and writes, sensitive to latency and IOPS, with comparatively modest requirements for sustained large-block read/write throughput.

10.1.2 Centralized vs. Distributed Block Storage

Centralized block storage (SAN): The traditional enterprise solution is a dedicated storage area network (SAN). An expensive, high-performance centralized storage array (such as products from EMC or NetApp) provides block devices to servers over Fibre Channel or iSCSI.

Advantages: high performance, mature and feature-rich.

Disadvantages: high cost, scale-up bottlenecks, proprietary and complex management — a classic "siloed" architecture.

Distributed block storage: Modern cloud environments prefer distributed block storage. It adopts a scale-out architecture, organizing the local drives of hundreds or thousands of commodity x86 servers, via software, into a single unified and enormous storage resource pool.

10.1.3 Open Source Implementation: Ceph RBD (RADOS Block Device)

Ceph is today's most popular and most feature-complete open-source distributed storage system. It is a unified storage platform that can serve block storage (RBD), object storage (RGW), and file storage (CephFS) from the same cluster. We begin by dissecting its block storage component — RBD.

Ceph's Core Architecture: RADOS

At the base of Ceph lies a highly reliable distributed object store called RADOS (Reliable, Autonomic Distributed Object Store). Understanding RADOS is the key to understanding every service that Ceph builds on top of it.

OSD (Object Storage Daemon): A Ceph cluster is made up of many storage nodes, each running multiple OSD daemons. Every OSD process directly manages one physical disk. The storage and retrieval of data is ultimately carried out by the OSDs.

Object: Within RADOS, all data — whether the upper layer treats it as blocks, files, or objects — is eventually subdivided into fixed-size objects (4 MB by default) for storage.

PG (Placement Group): To manage enormous numbers of objects efficiently, Ceph introduces the notion of a PG. A PG is a logical collection of objects. Objects are mapped into a PG through a hashing algorithm, and data distribution and replication are both performed at the granularity of PGs.

The CRUSH algorithm: This is Ceph's "soul." When reading or writing an object, how does the client know on which server and on which disk the object lives? Traditional distributed systems typically require a centralized metadata server to record this information, but such a server becomes a bottleneck. Ceph pioneered the CRUSH (Controlled Replication Under Scalable Hashing) algorithm. A client need only know the cluster's topology map (the CRUSH Map); through a series of hash computations it can independently and in real time derive the PG that corresponds to any object, and which OSDs hold that PG's primary and secondary replicas. This yields a design with no centralized metadata and confers exceptional scalability.

MON (Monitor): The MON is the cluster's "watchdog" and "arbiter." It does not store user data; it maintains only the cluster's critical state information, such as OSD liveness, the CRUSH Map, and authentication data. A Ceph cluster typically runs 3 or 5 MONs to form a highly available Paxos quorum.

How RBD Works:

1. Create an RBD image: The administrator creates an RBD "image" on the Ceph cluster — for example, a 100 GB virtual disk.
2. Striping: Ceph logically slices this 100 GB volume into consecutive, fixed-size objects (4 MB by default), such as image_object.0000, image_object.0001, ...
3. Client mounting: On the GPU server, the RBD image can be accessed through the kernel RBD driver (krbd) or the user-space librbd library (used with QEMU/KVM). The client fetches the latest CRUSH Map from the MON.
4. I/O flow: When the operating system wants to write data at the 5 MB offset of this block device, the RBD client driver computes that this location corresponds to the object image_object.0001. It then uses the CRUSH algorithm to independently derive the PG to which image_object.0001 belongs, and which OSD is the primary for that PG. The client connects directly to this primary OSD and sends it the write request. Upon receiving the data, the primary OSD is responsible for replicating it to the other backup-replica OSDs of the PG. Only after all replicas have been written successfully does it acknowledge write completion to the client.

10.1.4 Fault Recovery in Distributed Block Storage

OSD failure: When an OSD process — or the disk or server hosting it — fails, the MON quickly detects its heartbeat timeout and marks it as down.

Every PG that contained the failed OSD enters a "degraded" state.

Ceph immediately launches a self-healing process. For each degraded PG, it selects a new, healthy OSD from the surviving replicas and replicas the lost data onto it, restoring the PG to a full, healthy replica count. The whole operation is fully automatic and requires no human intervention.

10.1.5 Performance Optimization for Distributed Block Storage

Use SSDs/NVMe as OSDs: For block storage scenarios that demand high IOPS and low latency (such as system disks), it is strongly recommended to use SSDs or NVMe drives as the OSDs' physical medium.

Journal/WAL disk: Before writing data, each Ceph OSD first records the write operation in a journal. This journal can be placed on higher-performance media (for example, using HDDs as data disks while a small NVMe SSD serves as a shared journal disk for all the HDD OSDs), which greatly accelerates write response.

Cache tiering: Ceph supports configuring a "cache tier" of high-speed SSDs alongside a "base tier" of large-capacity HDDs. Hot data is automatically promoted to the cache tier to speed up access.

Through Ceph RBD, we can provide the entire GPU cluster with a unified, highly available, and scalable pool of block storage, neatly solving the storage needs of operating systems, container persistent volumes, and VM disks.

10.2 Massive Unstructured Data Storage — Distributed Object Storage

The datasets required for AI training — especially images, video, audio, and text — are unstructured. Their characteristics are:

Enormous capacity: on the order of terabytes, petabytes, or even exabytes.

A vast number of files: potentially millions or even billions of small files.

Access pattern: typically write-once-read-many (WORM). Data is rarely modified after it is uploaded.

Access interface: a simple, standard access method is required to ease integration with the many data-processing tools and frameworks.

For this scenario, traditional file systems (whether local or NFS) struggle. Distributed object storage, by contrast, has become the ideal choice for storing such enormous bodies of unstructured data.

10.2.1 Core Concepts of Object Storage

Unlike file systems and block storage, object storage adopts an extremely flat, simple three-level data model:

1. Data: the raw content itself — the file.
2. Metadata: a set of key-value pairs describing the data, which may be system-predefined (such as size and creation time) or user-defined (such as label:cat, source:camera1).
3. Universally unique ID (UUID): every object in the storage system carries a unique ID.

Flat namespace: Object storage has none of the complex, hierarchical directory-tree structure of a file system. All objects are "laid flat" inside a logical container called a bucket. The access path of an object is typically Bucket_Name/Object_Key.

Access via API: The primary way to access object storage is not by mounting it as a local file system, but through simple, standardized RESTful HTTP APIs. The best known is the Amazon S3 (Simple Storage Service) API, which has become the de facto standard in the field. Users carry out upload, download, and management through HTTP verbs such as GET, PUT, and DELETE.

10.2.2 Advantages of Object Storage

Nearly unlimited scalability: The flat namespace and the absence of centralized metadata (many object storage systems borrow algorithms similar to CRUSH) allow object storage to scale horizontally to thousands of nodes, managing exabyte-level data and trillions of objects.

Extremely low cost: Object storage is typically engineered to run on cheap, high-capacity SATA drives and commodity x86 servers, so the per-unit storage cost is very low. It also supports technologies such as erasure coding, which can guarantee data reliability at lower redundancy than multi-replication, further cutting costs.

High durability: Through multi-replication or erasure coding, extremely high data durability (for instance, eleven nines or better) can be achieved.

Standardized interface: The HTTP-based S3 API is simple and easy to use, and enjoys an extremely broad ecosystem. Virtually every big-data tool, AI framework, and client library can interact with S3-compatible storage.

10.2.3 The First Choice for Entry-Level Object Storage: Ceph RGW (RADOS Gateway)

Ceph makes another appearance. RGW is an S3-compatible object storage gateway provided by Ceph, built on top of the RADOS foundation.

Working principle:

RGW is a stateless daemon; multiple instances can be deployed to achieve high availability and load balancing.

When an S3 client (for example, a data-preprocessing script) sends a PUT request to upload an object, the request first reaches an RGW gateway. RGW handles S3-protocol authentication, authorization, and the like.

RGW then splits the uploaded object into multiple RADOS objects and, just like RBD, uses the CRUSH algorithm to compute which OSDs these objects should be stored on.

Finally, RGW communicates directly with those OSDs to complete the write.

Advantages: If you have already deployed a Ceph cluster for block storage, adding object storage is almost "free" — you need only start a few RGW processes. It offers the convenience of a unified storage platform.

Disadvantages: RGW's performance and its efficiency at S3 metadata operations may not be optimal compared with systems purpose-built for object storage.

10.2.4 Open Source Massive Object Storage: Swift

Swift is the object storage component of the OpenStack open-source cloud computing project. It is one of the earliest and most influential open-source object storage systems.

Architectural feature: the consistent hashing ring

Rather than computing dynamically as CRUSH does, Swift maintains a static "ring." This ring is a vast hashing space, and all storage nodes (drives) are mapped to distinct positions around it.

When an object is to be stored, its name is hashed to find a position on the ring, and the next N nodes clockwise (for example, 3) are chosen to hold replicas of the object.

Advantages: a simple architecture that is easy to understand and implement.

Disadvantages: mutations to the ring (such as adding or removing nodes) are relatively complex and time-consuming, and its scalability and flexibility fall short of Ceph's.

Current status: Swift remains widely used within the OpenStack ecosystem, but in newer projects its luster has been somewhat overshadowed by Ceph and MinIO.

10.2.5 Commercial Object Storage: Privatizing Public-Cloud Object Storage

The major public cloud vendors (AWS, Google Cloud, Azure, Alibaba Cloud, and others) all offer world-class, highly mature object storage services. In recent years they have also begun packaging the object storage technologies validated in their public clouds into software-hardware appliances or pure-software forms, offering enterprises the option of deploying them privately in their own data centers.

Advantages: the same experience, features, and enterprise-grade support as the public cloud.

Disadvantages: high cost, a typically closed-source technology stack, and the risk of vendor lock-in.

10.2.6 A Rising Star: RustFS

RustFS is an emerging open-source object storage project (Apache 2.0 licensed) positioned for "ultra-fast + cloud-native" workloads, rewritten from scratch in Rust, with the community benchmarking it against MinIO (see the project repository). The features below are all vendor self-descriptions that still lack independent third-party verification; readers should run their own benchmarks when evaluating.

Core features (as claimed by the project):

Built for extreme performance: built throughout on async/await + Tokio, with zero-copy path optimizations for NVMe-oF, RDMA, and 100 GbE. The project's own comparison claims about 2.3x MinIO's throughput for 4 KB small objects — a vendor/community self-test, not independently reproduced, and small-object performance does not extrapolate to large-object sequential read/write workloads.

Cloud-native design: a single binary plus stateless operation, with a K8s Operator and Helm charts, supporting containerized deployment and automatic scaling.

S3 compatibility: claims compatibility with mainstream S3 APIs (including encryption, Multipart, Versioning, Lifecycle, etc.); see the project documentation for the current pass rate.

Simple architecture: metadata and data are stored together on the same disks, removing the dependency on a standalone metadata database. To scale horizontally you simply add nodes; the system rebalances automatically.

Applications in AI (a synthetic example illustrating the architectural pattern, not measured results):

With its memory-level latency and Rust's memory safety, RustFS can serve as a "hot data cache layer." A typical pipeline:

1. Cold data is stored in a low-cost HDD Ceph or a public-cloud archive tier.
2. Before training, rustfs-cli preload asynchronously pulls the subset needed for the current epoch into the NVMe RustFS cluster.
3. GPU nodes read it directly with high concurrency through the S3 API. The gains of this "cold-hot tiering plus prefetch" pattern depend on dataset size, network bandwidth, and access locality, and must be validated by benchmarking the actual workload.

10.3 AI Training Data Storage — Distributed Concurrent High-Performance Storage

We have found suitable "homes" for the system and for the vast raw data. Now we confront the most formidable challenge of the entire storage design: how to "feed" the hundreds or thousands of GPUs engaged in large-scale distributed training.

The I/O pattern in this scenario is highly distinctive:

Massively concurrent reads: thousands of data-loader processes across hundreds of compute nodes may read the same set of training files — for instance, the data of a single epoch — at nearly the same instant. This puts enormous concurrent pressure on the storage system, giving rise to a so-called "read storm."

High throughput requirements: To keep GPUs from idling, the storage system must sustain aggregate read bandwidth that matches the total network bandwidth of all GPU nodes. A cluster of 128 DGX A100 nodes, for example, may have a total storage network bandwidth as high as 128 * 200 Gbps = 25.6 Tbps = 3.2 TB/s!

Performance on small files: Many datasets (such as ImageNet) consist of millions of small files. The storage system must handle the metadata operations and data reads for these small files efficiently.

POSIX compatibility: Most AI frameworks and data-loading libraries are still accustomed to accessing data through the standard POSIX file interface (open, read, seek).

For this "many-to-one" high-concurrency, high-performance read scenario, traditional NFS quickly becomes a bottleneck because of its centralized metadata server. Distributed object storage, though scalable, is ill-suited to serve as the hot-data layer of training, owing to its HTTP-based, high-latency access and its eventual consistency model.

The genuine solution is the distributed parallel file system, born for HPC.

10.3.1 The Core Idea of Parallel File Systems: Separating Metadata from Data

The central architectural idea of a parallel file system is to separate the file system's two major functions — metadata management and data storage.

Metadata Server (MDS): One or more dedicated servers that handle all metadata operations. These include creating, deleting, and renaming files and directories, checking permissions, retrieving file attributes (stat), and — most importantly — telling the client where a given file's data blocks are stored.

Object Storage Server (OSS / Data Server): A large body of servers responsible for the actual data storage. Each OSS manages several object storage targets (OSTs), that is, physical drives. A file's data is split into stripes and stored in round-robin fashion, in parallel, across many OSSs and OSTs.

Client: the file system client running on the compute node.

I/O flow:

1. Open operation: To open a file, the client first sends a request to the MDS.
2. The MDS performs a permission check, then looks up the file's "layout information" in its metadata store — that is, across which OSSs' which OSTs the file's data has been striped.
3. The MDS returns this layout information to the client.
4. Read/write operations: When the client next reads or writes file data, it no longer needs to contact the MDS. Acting on the layout it obtained, it connects directly and in parallel to all the relevant OSSs and reads (or writes) data stripes from multiple OSSs simultaneously.

Advantages:

Concurrency and bandwidth aggregation: By striping data across a large number of OSSs, multiple clients can access different OSSs in parallel, or a single client can read from several OSSs at once. The file system's aggregate bandwidth is the sum of all OSS bandwidths and scales linearly as more OSSs are added.

Metadata performance: By centralizing metadata operations on dedicated MDSs, the MDS can be specially optimized (with high-speed NVMe, large memory, and so on) to absorb the metadata pressure of massive numbers of small files. The MDS can also be configured as a high-availability cluster.

10.3.2 The Forebear of Open Source Big Data Storage: HDFS

HDFS (Hadoop Distributed File System) is the distributed file system of the Hadoop ecosystem. Although it is not a strictly POSIX-compatible parallel file system, its design ideas have exerted a profound influence on what came after.

Architecture: a centralized NameNode (responsible for all metadata) plus a large number of DataNodes (responsible for storing data blocks).

Characteristics: optimized for large files and streaming reads; write-once, read-many.

Limitations in AI:

The NameNode is a single point of bottleneck; metadata performance is limited, and it does not handle vast numbers of small files well.

It is not fully POSIX-compatible, requiring access through dedicated APIs or FUSE, which is unfriendly to existing AI applications.

10.3.3 Industry Improvements to HDFS

To overcome HDFS's metadata bottleneck, the industry has produced many improvements, such as Alluxio (a memory-first virtual distributed file system that can serve as a caching layer above HDFS) and commercial variants from vendors that support multiple metadata nodes.

10.3.4 The Evergreen: Lustre

Lustre is one of the most widely deployed and longest-lived open-source parallel file systems in the HPC field. Many of the world's fastest supercomputers use Lustre as their primary storage system.

Architecture: a classic metadata/data separation.

MDS (Metadata Server): responsible for metadata.

OSS (Object Storage Server): responsible for data.

Client: the Lustre client is a kernel module providing full POSIX compatibility.

Characteristics:

Extreme performance and scalability: Lustre is designed to support hundreds of thousands of clients and exabyte-level storage, delivering terabyte-per-second aggregate bandwidth.

Maturity and stability: after decades of development and service in top-tier supercomputing centers, Lustre is highly mature and stable.

Integration with AI: Lustre pairs perfectly with GPUDirect Storage. A Lustre client can write data directly from an OSS into GPU memory over an RDMA network, achieving end-to-end high-performance data loading.

Challenges:

Complex deployment and management: deploying and tuning a large-scale Lustre cluster demands a very high level of skill from operations staff.

10.3.5 Another Giant: IBM Spectrum Scale (formerly GPFS)

GPFS is IBM's commercial parallel file system, on a par with Lustre and widely used in both HPC and commercial settings.

Characteristics:

Decentralized architecture: architecturally, GPFS goes a step beyond Lustre. It has no centralized metadata bottleneck like Lustre's MDS; metadata management is distributed across many nodes in the cluster, giving better scalability.

Rich features: it offers abundant enterprise-grade functionality, including snapshots, multi-tier storage, and cloud integration.

Commercial product: GPFS is commercial software and requires a license.

10.3.6 How to Choose Training Data Storage?

For a top-tier AI supercomputing center that pursues ultimate performance and is unconstrained by cost or operational complexity, Lustre or GPFS is the natural choice.

For many enterprise-grade AI platforms, some easier-to-deploy and easier-to-manage commercial parallel file systems (such as WekaIO or DDN EXAScaler) are also worth considering, as is combining high-performance NVMe CephFS/MinIO clusters with upper-layer data caching or prefetching solutions (such as Alluxio) to serve as training storage, striking a balance between cost and performance.

10.4 Chapter Summary

In this chapter we carried out a comprehensive architectural design for the "data life" of the GPU cluster — its storage system. We came to appreciate that a single storage solution cannot satisfy the diverse needs of an AI workflow, and that a layered, heterogeneous storage architecture is the best practice.

We divided cluster storage into three logical tiers according to data type and access pattern, and matched each tier with the most suitable technology:

1. For program and system storage, we chose distributed block storage exemplified by Ceph RBD. By pooling the drives of ordinary servers, it provides operating systems, containers, and VMs with high-availability "cloud disks" that feel like local drives, resolving the persistence needs of systems and applications themselves. We delved into its decentralized design grounded in RADOS and the CRUSH algorithm, and understood the sources of its high availability and self-healing.
2. For the vast unstructured raw datasets, we chose distributed object storage exemplified by Ceph RGW and MinIO. We learned that object storage, with its flat namespace, S3-compatible API, near-unlimited scalability, and extremely low cost, is the ideal "data lake" for storing petabyte-scale training data.
3. For the most exacting scenario — concurrent access to AI training data — we focused on the distributed parallel file systems born for HPC. We dissected their core architecture of separating metadata from data, and understood how, by striping data across a great many data servers, they achieve astonishing concurrent read capability and aggregate bandwidth, capable of "feeding" hundreds or thousands of hungry GPUs. We introduced the two classics of the field — Lustre and GPFS — and recognized them as the unmistakable choice for building the storage foundation of a top-tier AI training cluster.

Through this layered storage architecture, we have built for the GPU cluster a data foundation that is functionally complete and balanced in performance:

Cold data layer / data lake: low-cost distributed object storage (such as HDD-based Ceph) to archive the massive raw datasets.

Warm data layer / hot data layer: high-performance distributed parallel file systems (such as NVMe-based Lustre) to host the data subsets currently being accessed with high concurrency by training jobs.

System and application layer: highly available distributed block storage (such as SSD-based Ceph RBD) to host operating systems and container persistent volumes.

This storage system works in close concert with the computing and network architectures we designed earlier. In particular, through technologies like GPUDirect Storage, it opens up the "last mile" that runs straight from the storage array into GPU memory, jointly forming an efficient, bottleneck-free data infrastructure truly built for the era of large models. In the chapters that follow, we will build upon this foundation to explore how to construct the higher-level application development and runtime platform.