祝融说。

In the previous chapters, we invested enormous effort in building a powerful AIDC "warship." We mastered its engine (GPU/NPU), became familiar with its sea lanes (networking and storage), understood its navigation (scheduling and parallelism), and even practiced damage control procedures for various emergencies. Now, it is time to appoint a "captain" for this warship and formulate a set of "battle doctrines" for its long voyage.

The "Two-Tier AIDC" -- the collaborative architecture of "centralized training and edge inference" -- is the core production model we designed for this intelligent computing factory. However, a successful system is far more than a physical division; it is a smooth, automated, and measurable operational workflow. If a new model trained at the center takes a week to deploy to edge nodes, or if the amount of compute used by business units becomes an indecipherable "messy account," then all our previous technical investments will be severely undermined.

In this chapter, as "system designers" and "operations planners," we will tackle two crucial issues. First, we will design a complete model distribution and synchronization mechanism, bridging the "last mile" from the center to the edge, achieving a closed loop in the model lifecycle. Next, we will delve into a highly challenging and valuable topic within enterprises -- the compute billing model -- learning how to scientifically price expensive AI compute and establish a fair, transparent internal settlement system. This concerns not only cost control but also cultivating a culture of "cherishing compute power and using clouds efficiently" throughout the enterprise.

11.1 Center-Edge Orchestration: Model Distribution Mechanism and Image Registry Synchronization

In the "Two-Tier AIDC" system, the central training center is the "model factory," and the edge inference nodes are the "product showrooms." Connecting the two, ensuring new products can be delivered quickly, safely, and reliably to the showroom, is the Model Distribution pipeline. A modern model distribution mechanism must possess characteristics like automation, versioning, security, and efficiency. It is essentially a CI/CD (Continuous Integration/Continuous Deployment) pipeline for models, the core artery of the LLMOps system.

11.1.1 The Challenge: From "USB Copy" to "Global Synchronization"

In the early stages of a project, model distribution might be very primitive: an algorithm engineer trains a model, packages it into a tens-of-GB compressed file, and manually copies it to the inference server via FTP, object storage, or even a USB drive. The drawbacks of this approach are obvious:

• Inefficient: Pure manual operation, time-consuming, error-prone.
• Version Confusion: Lack of strict version control, making it easy for the model version running online to not match the code and data.
• Poor Security: Model files could be tampered with or leaked during transfer.
• Cannot Scale: When inference nodes grow from one server to hundreds or thousands across multiple geographic regions, manual distribution becomes an impossible task.

11.1.2 Designing a Modern Model Distribution Pipeline

A standardized model distribution pipeline typically consists of three main components: the Model Registry, the CI/CD Engine, and the Image Registry. They work together to complete the transformation from "model checkpoint" to "online service."

Model Registry: The Model's "ID Card and Archive"

The Model Registry is the starting point and the "Single Source of Truth" for the entire pipeline. It is not a simple file store, but a system that performs version management and metadata recording for models. MLflow Model Registry and Hugging Face Hub are excellent representatives.

Core Functions:

• Versioned Registration: When a training task (whether pre-training or SFT) completes successfully, its output model files (weights, configuration, etc.) are registered in the Model Registry and assigned a unique version number (e.g., Llama3-8B-SFT-v2.1.0).
• Metadata Management: Accompanying the model are rich metadata:
  • Traceability: Which training script, code version, dataset, and hyperparameters produced this model?
  • Performance Metrics: What is its evaluation score (e.g., Accuracy, BLEU)?
  • Environment Dependencies: Which versions of PyTorch, CUDA does it depend on?
• Lifecycle Stage Management: A model version can be marked with different stages, such as Staging (testing), Production (ready for use), and Archived. This provides workflow assurance for gradual rollout and safe deployment.

CI/CD Engine: The Automated "Model Packaging Factory"

The CI/CD Engine (e.g., Jenkins, GitLab CI, Tekton) is the "orchestration hub" of the entire pipeline. It monitors the state changes in the Model Registry and automatically triggers a series of build, test, and deployment tasks.

Pipeline Trigger:

A webhook can be configured. When a new model version in the Model Registry is promoted from Staging to Production, it automatically triggers the CI/CD pipeline.

Core Build Stages:

• Model Optimization and Conversion:
  • Pull Model: The first step of the pipeline is to pull the new model file marked as Production from the Model Registry.
  • Quantization/Pruning: Call quantization tools (like AutoGPTQ, AWQ) to convert the FP16 model to INT8 or INT4, reducing size and improving inference speed.
  • Compile to Engine Format: If using TensorRT-LLM or MindIE as the backend, this step calls the corresponding compiler to compile the model into .engine or .om format optimized files.
• Image Building:
  • Package the optimized model files from the previous step, along with a lightweight inference server (e.g., a wrapper for vLLM or Triton), into a new Docker image.
  • The Dockerfile for this image should be written following the best practices from Chapter 4, ensuring it is lightweight and secure.
  • The image tag should be strongly associated with the model version, e.g., my-registry/llama3-8b-sft-service:v2.1.0.
• Image Pushing:
  • Push the newly built image to the central image registry (e.g., Harbor, Artifactory).

Image Registry Synchronization: The "Logistics Network" Connecting Center and Edge

When inference nodes are distributed across data centers or edge sites globally, having all nodes pull images from the central image registry is inefficient and unreliable. A layered, distributed image registry system is needed.

Architecture Design:

• Central Registry (HQ Registry): Located at the central data center, it is the authoritative source for all images.
• Regional/Edge Registry: Deploy a "pull-through cache" or "read-only replica" of the image registry in each major geographic region or edge data center.
• Synchronization Mechanism: Use the image registry's built-in replication feature. For example, in Harbor, replication rules can be configured:
  • Source: The production-models project in the central registry.
  • Target: All edge registries.
  • Trigger: Event-based. When a new image is pushed to the central registry, the sync task is automatically triggered immediately, distributing the image to all edge registries.

Workflow:

1. The CI/CD pipeline pushes the newly built ...:v2.1.0 image to the central registry.
2. The event trigger in the central registry is activated, starting replication tasks to all edge registries.
3. The image is efficiently synchronized to each edge node via dedicated lines or the public internet.
4. When the Kubernetes clusters at edge sites deploy a new model, they are configured to pull images preferentially from the local edge registry. This greatly accelerates deployment speed and reduces reliance on the central site's egress bandwidth.

Edge Deployment

When the image is synchronized to the edge registry, the central CD system (e.g., ArgoCD) updates the deployment configuration (like the image tag in the Deployment) of the corresponding service in the edge K8s cluster via GitOps, thereby triggering a rolling update and smoothly bringing the new model online.

Summary: The Model Distribution "Highway"

1. Start: An algorithm engineer marks a model version as Production in the Model Registry.
2. Trigger: The CI/CD engine detects the event and automatically starts the pipeline.
3. Processing: The pipeline optimizes, compiles the model, and packages it into a standardized Docker image.
4. Storage: The new image is pushed to the central image registry.
5. Logistics: The image registry's replication mechanism is triggered, automatically synchronizing the new image to edge registries worldwide.
6. Launch: The GitOps system updates the deployment configuration in the edge cluster, triggering a service rolling update, bringing the new model online.

This automated system can reduce a manual process that might take days down to the order of tens of minutes (depending on pipeline design and image size). It is the core guarantee for achieving agile LLMOps and rapidly iterating model capabilities.

11.2 Compute Billing Models: How to Design Internal Settlement Pricing (Per GPU-Hour vs. Per Token)

The construction and operation costs of an AIDC are staggering. A single H800 GPU can cost hundreds of thousands of RMB, and the annual electricity bill for a thousand-card cluster can exceed ten million RMB. Without establishing a scientific cost accounting and internal settlement mechanism, compute resources will quickly be abused and wasted, eventually becoming a heavy burden on the enterprise.

The purpose of designing an internal billing model is not just to "allocate costs," but to use pricing leverage to guide users (internal business units or algorithm teams) to use compute more efficiently and economically. A billing model shapes internal resource-allocation behavior; it does not guarantee any business returns.

Currently, there are two main types of internal billing models: "infrastructure" perspective time-based billing, and "service" perspective usage-based billing.

11.2.1 GPU-Hour Billing: Simple and Direct "Reservation" Model

This is the most basic and easiest to implement billing model. It treats GPUs and NPUs as resources similar to cloud servers, charging based on "resource type * quantity * duration."

• Billing Formula: Cost = Unit Price (RMB/GPU-Hour) * GPU Count * Usage Duration (Hours)
• How to Determine the "Unit Price"? -- The TCO Method The formulation of the "unit price" is the core of this model. A reasonable unit price should cover the Total Cost of Ownership (TCO) of the GPU over its entire lifecycle.
  1. Hardware Cost Amortization:
     • C_hw = (Server Cost + GPU Cost + Network Equipment Cost per Port) / (Depreciation Years * 365 * 24)
     • Server cost includes CPU, memory, hard drives, etc.
     • Network equipment cost needs to be apportioned per port.
     • Depreciation period is typically 3-5 years.
  2. Power Cost:
     • C_power = (GPU Power + Other Server Component Power) * PUE * Electricity Price
     • PUE (Power Usage Effectiveness): The energy efficiency metric for data centers, typically between 1.2 and 1.5. PUE=1.3 means that for every 1 kWh consumed by IT equipment, the data center consumes a total of 1.3 kWh (the extra used for cooling, lighting, etc.).
     • Electricity price needs to consider peak/off-peak rates.
  3. IDC Colocation Cost:
     • C_idc = (Rack Rental + Bandwidth Fee) / (GPUs per Rack * 24 * 30)
     • This part is typically calculated based on the monthly rack cost, then apportioned per card.
  4. Personnel and Software Cost:
     • C_sw_om = (AI Infra Team Labor Cost + Commercial Software License Cost) / (Total Cluster GPUs * 24 * 30)

Final Unit Price (RMB/GPU-Hour) = (C_hw + C_power + C_idc + C_sw_om) * (1 + Profit Margin)

Profit Margin: Even for internal settlement, a 10-20% "profit margin" or "resource pool development fund" is typically added for future technology upgrades and capacity expansion.

Data Collection and Billing:

• Data Source: K8s scheduling logs (Pod creation and destruction times), Device Plugin allocation records, Volcano/Yunikorn job events.

• Process:

  1. Periodically (e.g., every hour), scan all running Pods that have requested GPU resources.
  2. Record each Pod's namespace (representing the business unit), pod_name, number of GPUs requested, and Pod lifecycle.
  3. At the end of the month, aggregate the "GPU-Hour" consumption for each namespace, multiply by the unit price, and generate the bill.

• Advantages:

  • Simple to implement: Data sources are clear, logic is straightforward.
  • Stable cost recovery: As long as a card is allocated, revenue is generated, facilitating financial forecasting.

• Disadvantages:

  • Cannot reflect actual usage efficiency: A user who requests 8 cards for 10 hours but has poorly written code with only 10% GPU utilization (HFU) pays the same as a user achieving 90% GPU utilization. This does not incentivize users to optimize their programs.
  • Not suitable for inference services: Inference services are shared, with multiple users using them simultaneously. Simply charging by "occupancy" is not feasible.

11.2.2 Token-Based Billing: A Refined "Pay-Per-Use" Model

This model is mainly applied to billing for inference services. It is entirely based on "business value" -- users pay for what they use. This is exactly the same billing method used by public cloud services like OpenAI.

• Billing Formula: Cost = Input Token Unit Price * Total Input Tokens + Output Token Unit Price * Total Output Tokens

• Why separate pricing for input and output?

  • Processing input (Prompt Processing) and generating output (Decoding) are computationally different.
  • Typically, processing input is more compute-intensive (parallel processing) but done once; generating output is serial but done over many steps.
  • Separate pricing allows for a more accurate reflection of the cost structure and encourages users to optimize their prompts (e.g., using shorter, more effective prompts).

• How to Determine the "Unit Price"? -- Reverse Cost Calculation The token unit price derivation is a "reverse engineering" process, combining cost, throughput, and expected profit.

  1. Calculate per-GPU-hour cost (Cost_per_GPU_Hour):
     • Use the TCO unit price calculated in Section 11.2.1. Assume it is 10 RMB/GPU-Hour.
  2. Stress test to get per-GPU performance (TPS_per_GPU):
     • Use tools like Locust to stress test the inference service of the target model deployed on a single GPU (see Section 8.3).
     • Find the maximum sustainable token generation rate (TPS) while meeting the SLA (e.g., 99% TTFT < 500ms). Assume for a Llama 3-8B model, stress testing reveals a single A800 can achieve 500 tokens/sec.
  3. Calculate cost per token (Cost_per_Token):
     • Cost per GPU per second = Cost_per_GPU_Hour / 3600 = 10 / 3600 = 0.00278 RMB/s
     • Cost to generate 1 token = Cost per GPU per second / TPS_per_GPU = 0.00278 / 500 = 0.00000556 RMB
  4. Set final selling price (Price):
     • Price_per_Token = Cost_per_Token * (1 + Profit Margin)
     • Price_per_1K_Tokens = 0.00000556 * 1000 * 1.2 (20% profit margin) = 0.0067 RMB
     • For market comparison, it is usually expressed in "RMB per million tokens." 0.0067 * 1000 = 6.7 RMB / million tokens.

Data Collection and Billing:

• Data Source: Access logs generated by the inference service's API gateway or the service itself. Each request log must contain user identity, input token count, and output token count.

• Process:

  1. All inference requests pass through a unified API gateway.
  2. The gateway is responsible for authenticating user identity and recording the token details for each call.
  3. This log data is sent in near real-time to a billing database or data warehouse.
  4. At the end of the month, aggregate token consumption by user ID, multiply by the unit price, and generate the bill.

• Advantages:

  • Fair and Precise: Charges based entirely on actual usage, which is the fairest for users.
  • Incentivizes Optimization: Encourages users to use shorter prompts or choose smaller models for their tasks, naturally guiding cost optimization.
  • Tied to Business Value: The billing model is directly linked to the number of business calls, making it easy for business units to understand and budget.

• Disadvantages:

  • Complex to Implement: Requires building a complete system for API gateway, authentication, log collection, and billing data processing.
  • Difficult Pricing: Unit price formulation depends on accurate stress testing and requires creating a complex price list based on different models and hardware.

11.2.3 Hybrid Model: The Best Practice for a Two-Tier Operations System

In the "Two-Tier AIDC" system, a single billing model is insufficient. The best practice is to adopt a hybrid billing model:

• For the Central Training Center: Use "GPU-Hour Billing"
  • Training tasks are characterized by long duration and exclusive use of many GPUs. GPU-Hour billing best matches this resource usage pattern.
  • Advanced Feature: Introduce utilization-based penalties and rewards.
    • The GPU-Hour unit price can be variable. For example, if a user's average job HFU is below 20%, their GPU-Hour unit price is increased by 20% as a "resource waste penalty." If their average MFU is above 50%, their unit price is reduced by 10% as an "efficient usage reward."
    • This requires deep integration of the billing system with the monitoring system, but it greatly incentivizes users to optimize their training tasks.
• For Edge Inference Nodes: Use "Token-Based Billing"
  • Inference services are shared, multi-tenant online services. Token-based billing is usually the internal settlement method best matched to this usage pattern (alternatives such as per-request billing also exist).
  • The AI Infra platform team acts as the "provider" of inference services, settling internally with the business units that call the service.
  • Business units do not need to care about how many underlying cards were used; they only pay for the API calls they made.

Summary:

Compute billing is a key step for AIDC operations to move from "technology" to "business management." It translates technical costs in a measurable and understandable way to the ultimate business users. By designing a hybrid billing model -- primarily "GPU-Hour billing," supplemented by "Token-Based billing," combined with utilization-based penalties and rewards -- the AI Infra team can not only effectively recover costs but also play the role of "compute optimization consultant," driving the entire organization's technology and culture towards "Lean AI Computing."