祝融说。

After pre-training, fine-tuning, and rigorous compute accounting, we finally have a high-performing large model. However, a model file sitting on a hard drive generates no value by itself. The real challenge now begins: how to deploy this massive model into an online service that can serve thousands of users simultaneously, with lightning-fast response times and controllable costs?

This is the problem that large model inference aims to solve. Unlike training, inference scenarios have extremely demanding requirements for latency and throughput. Users cannot tolerate waiting tens of seconds for each reply from a chatbot. At the same time, for enterprises, every GPU used for inference must serve as many users as possible to amortize the expensive hardware and power costs.

In this chapter, we will focus on the full-stack optimization of large model inference. We will first dive into the "three kingdoms" of today's inference engines, comparing and analyzing the strengths and weaknesses of the three major solutions: vLLM, TensorRT-LLM, and Huawei MindIE. Next, we will go deep into the "black magic" core of these engines, revealing the principles behind revolutionary technologies like PagedAttention and Continuous Batching. Finally, we will pick up stress-testing tools and manually stress-test a deployed model service, learning how to scientifically measure and evaluate its performance. Mastering this chapter will equip you with the ability to build enterprise-grade large model services, bridging the "last mile" from model to business.

8.1 The Battle of Inference Engines: vLLM, TensorRT-LLM, and MindIE (Ascend)

Directly using training frameworks like PyTorch or TensorFlow for online inference is a highly inefficient practice. Training frameworks, designed for flexibility and ease of use, contain many components unnecessary for inference (such as autograd and optimizers), and their memory management and execution models are not optimized for the characteristics of inference scenarios.

To address this problem, a series of inference engines designed specifically for large model inference have emerged. They are like specially built racing engines for F1 cars, discarding all unnecessary features and squeezing out every drop of performance.

8.1.1 The Plight of Naive Inference: Inefficient Memory and Computation

Let us first see what happens when using the transformers library directly for inference, without an inference engine.

• The KV Cache Memory Disaster:

  • A Transformer model, when generating each new token, needs to reference the Keys and Values of all previous tokens (the KV Cache).
  • The naive implementation pre-allocates a huge contiguous space for each request's KV Cache, capable of accommodating its maximum possible length (e.g., 4096).
  • Problem: If a user only inputs 10 tokens, the KV Cache space for the remaining 4086 tokens is left idle and wasted. For scenarios with a large number of short requests, this waste is staggering. An 80 GB A800 might, because its memory is filled with these "vacant" KV caches, only be able to serve a handful of requests simultaneously.

• The Inefficiency of Static Batching:

  • To leverage the parallel computing power of GPUs, a natural idea is to pack multiple requests into a batch for processing.
  • Problem: Traditional static batching requires all sequences in a batch to start and end simultaneously. This means that even if a request has finished generating, it must wait in place until the longest request in the batch is also complete before resources can be released and results returned. This causes the GPU to spend a significant amount of time "waiting," severely wasting compute resources and artificially prolonging user wait times.

The core goal of inference engines is to solve these two major pain points.

8.1.2 vLLM: The "Ease-of-Use King" Born for Python

vLLM is an open-source project initiated by researchers at UC Berkeley. With its revolutionary PagedAttention technology and excellent ease of use, it quickly became the first choice of academia and many startups.

• Core Technology: PagedAttention (detailed in Section 8.2)

  • vLLM's "killer feature." It borrows concepts of virtual memory and paging from operating systems, dividing the KV Cache from contiguous memory blocks into non-contiguous, fixed-size "Blocks."
  • This makes KV Cache management extremely flexible, achieving near-zero memory waste. It also allows efficient sharing of KV Cache between requests (e.g., for multiple requests sharing the same prefix), further saving memory.
  • The result is significantly higher throughput on the same hardware and under typical workloads: the vLLM paper reports up to 2-4x higher throughput than the state-of-the-art systems of the time (FasterTransformer and Orca), while the official vLLM blog gives figures of up to 14-24x (roughly an order of magnitude, in long-sequence scenarios) over the unoptimized HuggingFace Transformers baseline; the exact multiple depends on workload and parameter settings.

• Architecture and Ecosystem:

  • Python Native, Easy to Integrate: vLLM is written entirely in Python and seamlessly integrates with the HuggingFace ecosystem. You almost never need to modify model code; just a few lines of Python code are enough to start a high-performance inference service.
  • OpenAI-Compatible API Server: vLLM comes with a built-in HTTP server compatible with the OpenAI API format. This means you can call your own deployed model service just like you call the OpenAI API, greatly simplifying upper-layer application development.
  • Distributed Inference: Supports Tensor Parallelism, allowing large models to be deployed across multiple GPUs.
  • Streaming Output: Supports token streaming generation for fast response and improved user experience.

• Selection Considerations:

  • Advantages:
    • Excellent Performance: The throughput improvement from PagedAttention is tangible.
    • Extremely Easy to Get Started: Very developer-friendly for Python users, with a gentle learning curve.
    • Active Open Source: The community is very active, with fast support for new models and new hardware (e.g., AMD GPUs).
  • Disadvantages:
    • Quantization Support Lags Behind: Compared to TensorRT-LLM, vLLM's support for more extreme optimizations like low-bit quantization (e.g., INT4) is slower.
    • CUDA Kernel Optimization: While its PagedAttention kernel performs well, deep optimization for other operators may not match TensorRT-LLM, which is hand-crafted by NVIDIA.

8.1.3 TensorRT-LLM: NVIDIA's "Performance Nuclear Weapon"

TensorRT-LLM is NVIDIA's official solution for large model inference. It is built on top of the mature TensorRT (NVIDIA's high-performance deep learning inference SDK) and is the "ultimate choice" for pursuing extreme performance and low latency.

• Core Technology: Deep Compilation Optimization + In-Flight Batching

  • Compile-Time Optimization: The core idea of TensorRT-LLM is "compile first, run later." It converts your model (e.g., a PyTorch model) into a highly optimized TensorRT Engine. During this compilation, it performs a series of "black magic" operations:
    • Operator Fusion: Combines multiple small CUDA kernels (e.g., MatMul + Bias + ReLU) into a single large kernel, reducing kernel launch overhead and memory reads/writes.
    • Precision Calibration and Quantization: Supports low-precision inference like FP16, INT8, and even INT4, and can automatically calibrate for maximum performance while maintaining accuracy.
    • Hardware-Specific Kernel Selection: Automatically selects the optimal CUDA implementation based on your specific GPU model (e.g., H800).
  • In-Flight Batching: This is TensorRT-LLM's implementation comparable to vLLM's Continuous Batching. It allows dynamically adding new requests to and removing completed requests from a batch during inference, maximizing GPU utilization.

• Architecture and Ecosystem:

  • C++ Core, Performance First: Its core runtime is written in C++, with minimal performance overhead.
  • Integration with Triton Inference Server: TensorRT-LLM is typically deployed alongside NVIDIA's Triton inference server. Triton provides enterprise-grade serving management features such as dynamic batching, multi-model deployment, HTTP/gRPC interfaces, and performance monitoring.
  • Complex Build Process: Using TensorRT-LLM requires a distinct build step. You need to download the model from HuggingFace, compile it into an Engine file using TensorRT-LLM's Python API, and then use Triton to load these Engine files for serving.

• Selection Considerations:

  • Advantages:
    • Extreme Performance: Usually the industry benchmark for low latency and low-bit quantization.
    • Enterprise-Grade Features: Integration with Triton provides very complete and stable serving capabilities.
    • Official NVIDIA Support: Can fully exploit new features of NVIDIA hardware (such as FP8).
  • Disadvantages:
    • Complex to Use: Steep learning curve; requires understanding concepts like compilation and Engine building; troubleshooting is more difficult.
    • Poor Flexibility: Once the model is compiled into an Engine, it is fixed. Changing certain parameters (like maximum batch size) may require recompilation.
    • Relatively Closed Ecosystem: Tightly coupled with NVIDIA hardware and software stack.

8.1.4 MindIE (MindSpore Inference Engine): The Inference Power Tool for the Ascend Ecosystem

MindIE is the inference engine in the Huawei Ascend ecosystem, comparable to TensorRT-LLM. As part of the CANN software stack, it aims to provide extreme performance optimization for large model inference on Ascend chips (such as Ascend 310 and 910).

• Core Technology: Graph Optimization + Ascend Hardware Affinity

  • Graph-Level Collaborative Optimization: MindIE receives computation graphs from upper-layer frameworks (like MindSpore or PyTorch for Ascend) and performs a series of deep optimizations targeting the Ascend DaVinci architecture, such as operator fusion, memory reuse, and data format conversion.
  • Hardware Operator Acceleration: It maps key operators in the computation graph (like matrix multiplication) directly to the DaVinci architecture's 3D Cube for execution, maximizing hardware utilization.
  • Dynamic Batching Support: Supports mechanisms similar to Continuous Batching to improve throughput.
  • Quantization Support: Supports weight quantization to achieve lower latency inference on Ascend chips.

• Architecture and Ecosystem:

  • Deep Integration with CANN: MindIE is a native component of the Ascend software stack and can most directly and effectively utilize underlying hardware capabilities.
  • Service Deployment: Provides service deployment tools that package the optimized model into a callable online service.
  • Adaptation to Mainstream Models: Huawei is actively adapting MindIE to mainstream open-source large models like Llama, GLM, and Qwen, providing official conversion and deployment scripts.

• Selection Considerations:

  • Advantages:
    • Optimal Performance on the Ascend Platform: On Ascend hardware, MindIE typically provides better inference performance than other third-party frameworks (e.g., using PyTorch for Ascend directly).
    • Official Support: As the official solution, it offers Huawei technical support and continuous performance optimization.
  • Disadvantages:
    • Ecosystem Lock-in: Only works on the Huawei Ascend platform.
    • Community and Documentation: Compared to vLLM and TensorRT-LLM, the breadth and activity of its open-source community, as well as the richness of third-party documentation and tutorials, may still need development.

Comparison Summary:

Advice for AI Infra Engineers: On the NVIDIA platform, vLLM is the best choice for rapid startup and iteration, while TensorRT-LLM is the ultimate destination for pursuing extreme production performance. A common practice is to use vLLM during development and experimentation, and once the model and business logic are stable, invest engineering resources to migrate to TensorRT-LLM for final performance and stability. On the Ascend platform, MindIE is the natural first choice.

8.2 Core Technologies: KV Cache, PagedAttention, and Continuous Batching Principles

The performance improvements of inference engines do not appear out of nowhere; they stem from deep insights into the resource bottlenecks of large model inference and clever algorithmic design. In this section, we will delve into the core of these engines, understanding how they solve memory and computation efficiency problems.

8.2.1 KV Cache: The Cost of Memory

• Principle Review: The Transformer's self-attention mechanism is "context-dependent." When generating the i-th token, the model needs to recall and compute the attention relationship between the current token and all previous i-1 tokens. To avoid recomputing the Key and Value vectors for the previous i-1 tokens each time, a standard optimization is to cache them. This cache is the KV Cache.
• Memory Footprint Formula: Memory_KV_Cache (GB) = 2 * L * h * s * b * 2 (bytes/FP16) / 10^9 Where L is the number of layers, h is the hidden dimension, s is the sequence length, and b is the batch size.
• Pain Points:
  • Waste: As mentioned earlier, pre-allocating KV Cache for the maximum length for each request leads to a large amount of wasted memory.
  • Fragmentation: Different requests have different KV Cache sizes. When dynamically allocating and freeing, it easily creates many unusable "small fragments" in memory.

8.2.2 PagedAttention: Managing KV Cache Like CPU Memory

vLLM's PagedAttention technology is an elegant solution to the KV Cache problem.

Core Idea

1. Spatial Discretization (Paging): Instead of allocating contiguous large chunks of memory for the KV Cache, it is divided into many fixed-size, smaller blocks. Each block can store the Keys and Values of tens of tokens.
2. Logical Contiguity (Page Table): Maintain a page table for each request. This page table records which physical blocks should logically correspond to the request's token sequence.
3. On-Demand Allocation: When a request starts, the system only allocates one block for it. When the generated tokens fill this block, the system allocates the next block from a global block pool and updates the page table.

Revolutionary Advantages

• Significantly Improved Memory Utilization: Because small memory blocks are allocated on demand, internal fragmentation is nearly eliminated. The vLLM paper reports that KV-cache memory waste can be reduced to under 4% (i.e., over 96% utilization; from its comparative experiments, with actual results depending on workload shape).
• Efficient Sharing (Copy-on-Write): When multiple requests share the same prefix (e.g., multiple users all start with "Please summarize the book 'The Three-Body Problem'"), PagedAttention can direct their page tables to the same physical blocks storing that prefix's KV Cache. Only when a request starts generating different subsequent content does the system copy and allocate new blocks for it. In scenarios like parallel sampling and Beam Search, this dramatically saves memory and computation.
• Flexible Memory Management: Like virtual memory in operating systems, it can easily implement advanced operations like block swapping.

PagedAttention's emergence single-handedly elevated the throughput of large model inference by a notch. It is one of the most important innovations in the AI Infra field in recent years.

8.2.3 Continuous Batching: Keeping the GPU "Always Busy"

Having solved the memory problem, the next goal is to improve computation efficiency. Continuous Batching is designed for this purpose. Modern inference engines like vLLM, TensorRT-LLM, and MindIE all implement this technology, though the name may differ (e.g., In-Flight Batching).

Evolution of the Idea

1. Static Batching:
   1. Workflow: Gather a batch of requests -> Compute one step in parallel -> All requests generate one token -> Repeat.
   2. Disadvantage: The "bucket effect." Must wait for the slowest (longest) request in the batch to complete before the fast ones can proceed. The GPU spends a lot of time idle near the end.
2. Continuous Batching:
   1. Workflow: Maintain a continuously running, dynamic batch.
   2. Iteration Loop: In each iteration loop of the inference server:
      1. Check for Completion: Check if any requests in the current batch have finished generating (e.g., generated the [EOS] token, or reached the maximum length). Immediately remove these completed requests from the batch and return their results to the user.
      2. Add New Requests: Check the waiting queue for new requests. If the GPU's compute and memory resources are available (because completed requests were just removed), dynamically add new requests to the current batch.
      3. Execute One Step: Perform one step of forward propagation on this "updated" dynamic batch, generating the next token for all requests in the batch.
      4. Return to step a.

Value

• Maximized GPU Utilization: Through the strategy of "arrive and leave, dynamically add and remove," the GPU is ensured to be processing a "full" batch as much as possible at every computation step. Idle time ("bubbles") is greatly reduced.
• Lower Average Latency: Short requests no longer have to wait for long requests; they can complete and be returned quickly, significantly reducing the average waiting time for users.

Summary: PagedAttention optimizes memory from the "space" dimension, while Continuous Batching optimizes computation from the "time" dimension. The combination of these two technologies forms the performance foundation of modern large model inference engines.

8.3 Stress Testing in Practice: Using Locust to Test TTFT and TPOT

After deploying an inference service, how do we scientifically evaluate its performance? "It feels fast" is not reliable. We need data to speak. Stress testing is a core means of verifying inference service performance, discovering bottlenecks, and performing capacity planning.

8.3.1 Core Performance Metrics

In LLM inference scenarios, we need to focus on two core metrics:

• TTFT (Time To First Token):
  • Definition: The time elapsed from when a user sends a request to when they receive the first generated token.
  • Significance: Directly affects the user's sense of "instant response." For conversational applications, low TTFT is crucial -- it makes the system feel "alive." TTFT mainly includes network latency, request queue waiting time, and the model's prompt processing time.
• TPOT (Time Per Output Token) / Tokens per Second (TPS):
  • Definition: The average time to generate each subsequent token (TPOT), or its reciprocal -- how many tokens can be generated per second (TPS).
  • Significance: Reflects the model's "generation speed." For tasks requiring long text generation, high TPS means users can see the complete result faster. TPS is mainly determined by the computation speed of the model's single-step decoding.

In addition, we also care about throughput, which can be measured in requests per second (RPS) or total generated tokens per second.

8.3.2 The Stress Testing Tool: Locust

Locust is an open-source, easy-to-use, distributed stress testing tool written in Python. It is very suitable for testing LLM services because we can flexibly simulate user behavior using Python.

• Advantages:
  • Test Scripts in Python: Very intuitive, easy to write complex test logic.
  • Distributed: Can easily initiate load from multiple machines, simulating a large number of concurrent users.
  • Web UI: Provides a clean web interface for viewing real-time statistics on QPS, response times, failure rates, etc.

8.3.3 Practical Stress Testing Steps

Scenario: We have deployed a Llama 3-8B model using vLLM, with its API service at http://127.0.0.1:8000/v1/completions.

Install Locust

pip install locust

Write the Stress Test Script (locustfile.py)

This script defines the behavior of "virtual users."

import time
from locust import task, FastHttpUser
import random
import json

# Simulate some user inputs
PROMPTS = [
    "你好，请做个自我介绍。 ",
    "请写一首关于春天的五言绝句。 ",
    "Explain the theory of relativity in simple terms.",
    "What are the top 5 tourist destinations in Japan?",
]

class LLMUser(FastHttpUser):
    # host = "http://127.0.0.1:8000" # Can be specified at launch

    @task
    def generate_text(self):
        prompt = random.choice(PROMPTS)
        output_len = random.randint(50, 200) # Random output length

        payload = {
            "model": "Llama-3-8B",
            "prompt": prompt,
            "max_tokens": output_len,
            "temperature": 0.7,
            "stream": True # Use streaming interface to measure TTFT
        }

        headers = {"Content-Type": "application/json"}
        
        start_time = time.time()
        first_token_received = False
        total_tokens = 0
        
        with self.client.post("/v1/completions",
                                json=payload,
                                headers=headers,
                                stream=True,
                                name="/v1/completions/stream",
                                catch_response=True) as response:
            
            if response.status_code != 200:
                response.failure(f"Request failed with status {response.status_code}")
                return

            try:
                for chunk in response.iter_lines():
                    if chunk:
                        # Parse the streamed JSON response
                        decoded_chunk = chunk.decode('utf-8')
                        if decoded_chunk.startswith("data: "):
                            data = json.loads(decoded_chunk[6:])
                            
                            # Record TTFT
                            if not first_token_received:
                                ttft = (time.time() - start_time) * 1000 # ms
                                self.environment.events.request.fire(
                                    request_type="POST",
                                    name="TTFT",
                                    response_time=ttft,
                                    response_length=0,
                                )
                                first_token_received = True

                            # Count tokens
                            if "choices" in data and len(data["choices"]) > 0:
                                total_tokens += 1
            
            except Exception as e:
                response.failure(str(e))
        
        # After stress test ends, TPOT/TPS can be calculated, but Locust does not natively support this
        # Typically, data needs to be exported for post-analysis
        # Here we mainly focus on TTFT and RPS through the Locust UI

Key Point: We precisely measure TTFT by capturing the first data chunk of the streamed response and report it to Locust as a custom request type.

Start Locust

locust -f locustfile.py --host http://127.0.0.1:8000

Begin Stress Testing

• Open a browser and visit http://localhost:8089.
• Set the number of concurrent users and the spawn rate. For example, simulate 100 concurrent users, adding 10 per second.
• Click "Start swarming."

Analyze Results

• In the Locust Web UI, you can see in real time:
  • RPS (Requests per second): How many new requests your service can handle per second.
  • Response Time Statistics: In the "Charts" tab, you can see the response time distribution graph for the request type named TTFT, including average, median, 99th percentile, etc. The 99th percentile TTFT is an important indicator of service stability.
• Calculating TPS:
  • On the server side, you need to use monitoring (like Prometheus) to scrape metrics exposed by vLLM, such as vllm_generation_tokens_total.
  • TPS = rate(vllm_generation_tokens_total[1m]), i.e., the total number of tokens generated per second.
  • Throughput = TPS * RPS (choose the appropriate throughput definition for different scenarios).

Bottleneck Analysis During Stress Testing

• TTFT Too High:
  • Increases under high concurrency: Means requests are waiting too long in the queue. Need to add more GPUs or optimize prompt processing performance.
  • High even under low concurrency: Model loading or prompt processing itself may be slow.
• TPS Low:
  • Means single-step decoding is slow. Try using lower precision (e.g., INT8 quantization), or switch to a more powerful GPU.
• RPS Stalls:
  • After reaching a certain RPS value, TTFT sharply increases or the error rate rises. This is typically the service's performance inflection point.
  • At this point, combine server-side monitoring of GPU utilization, memory usage, etc., to analyze the bottleneck:
    • GPU utilization is maxed out: Congratulations, your service is well-optimized; the bottleneck is the compute power itself. To improve RPS, only adding more cards will help.
    • GPU utilization is low but RPS stalls: The bottleneck may be in the CPU (e.g., Python's GIL, API server I/O), network, or scheduling logic.

Through such a complete cycle of stress testing -> analysis -> optimization, you can truly refine a large model inference service into an enterprise-grade application capable of handling real-world complex traffic.