祝融说。

In the first two parts of this book, we devoted considerable effort to completing full-stack infrastructure construction from the micro to the macro level. We now possess top-tier GPU servers, high-speed non-blocking physical networks, flexible and secure virtual networks, and layered, heterogeneous high-performance storage systems. We have also mastered how to pool, schedule, and isolate underlying physical resources through virtualization and containerization technologies. By this point, we have built an "AI city" with complete infrastructure, well-developed transportation, and abundant energy.

Yet a city only comes alive once residents move in and business begins to operate. Starting with this chapter, we enter the third part of the book, shifting our perspective from "city builders" to "city operators" and "industry planners." Our core task is to build, atop this powerful "AI city," an efficient, convenient, and standardized application development and runtime platform that lets data scientists, algorithm engineers, and application developers easily "move in with just their luggage." They can then quickly develop, deploy, run, and manage their AI applications without concerning themselves with the complexity of the underlying infrastructure.

A modern AI platform is no longer a simple, primitive environment offering only Jupyter Notebooks and command-line terminals. It is a complex, cloud-native Platform as a Service (PaaS) composed of numerous microservices, middleware, and observability systems.

This chapter, "Design and Implementation of a Machine Learning Application Development and Runtime Platform," serves as the opening of Part 3 and focuses on building this critical PaaS. We will explore how to decouple a modern AI application from a sprawling monolithic program into a series of collaborative microservices, and how to provide these microservices with a powerful runtime foundation.

Following the path of building a complete application ecosystem, we will explore:

1. The Microservices Platform: This is the cornerstone of the entire application platform. We will delve into how the container orchestration platform centered on Kubernetes has become the operating system of the cloud-native era, and introduce service governance frameworks such as Spring Cloud and Istio that address complex issues like service discovery, load balancing, and circuit breaking among microservices.
2. Middleware Services: If microservices are the "business logic units" of an application, then middleware is the "public utility infrastructure" that connects and supports these units. We will focus on the three most important categories of middleware: message middleware (such as Kafka and RabbitMQ), cache middleware (such as Redis), and databases.
3. Application Logging Services: In a distributed system composed of hundreds or thousands of microservices, logs serve as the "black box" for troubleshooting, monitoring, and auditing. We will learn how to build a centralized, searchable logging service system, such as the classic EFK (Elasticsearch, Fluentd, Kibana) architecture.

By the end of this chapter, you will be able to design a complete, production-grade AI application platform from the perspective of an application architect. You will master core concepts such as microservices, service mesh, message queues, and distributed caching, and understand how to combine them organically to provide a stable, efficient, scalable, and easily manageable modern development and runtime environment for your AI models and applications, whether for training, inference, or data processing.

11.1 The Microservices Platform

In the early days of AI application development, many projects existed as monolithic applications. For example, a complete recommendation system might be a single enormous Python or Java process, internally containing all functional modules: user request handling, feature engineering, model loading, online prediction, result ranking, logging, and so on.

This monolithic architecture offers fast initial development and easy deployment. However, as business complexity grows and team size expands, its drawbacks become increasingly apparent:

Rigid Technology Stack: All modules must use the same programming language and technology stack.

Reduced Development Efficiency: Any minor change can affect the whole system, requiring regression testing and redeployment of the entire application, resulting in long release cycles.

Poor Scalability: A performance bottleneck module (such as the model prediction module) cannot be horizontally scaled independently; instead, the entire large application must be duplicated.

Poor Reliability: A memory leak or bug in any single module can crash the entire application.

Difficult Onboarding for New Developers: The enormous codebase and intricate internal dependencies make it hard for new team members to understand the system quickly and contribute.

To address these problems, the microservices architecture emerged. Its core idea is to decompose a large monolithic application, along business boundaries or functional domains, into a series of small, independent, loosely coupled services. Each service is responsible for a single, well-defined function; it has its own independent codebase and data storage, and can be developed, deployed, and scaled independently. These services communicate through lightweight, well-defined APIs (typically HTTP RESTful APIs or gRPC).

Building and managing a complex system composed of hundreds or thousands of microservices requires a powerful microservices platform as its foundation. The heart of this platform is the "operating system" of the cloud-native era: Kubernetes.

11.1.1 Kubernetes: The Microservice Foundation Platform

In Chapter 7, we introduced how Kubernetes schedules GPU containers through Device Plugins. Now, from a more macroscopic perspective, we will understand how Kubernetes provides the foundational deployment, scaling, and self-healing capabilities for an entire microservices architecture.

Kubernetes' Core Abstractions:

Pod: The smallest deployment and scheduling unit in K8s. A Pod can contain one or more closely related containers that share the same network namespace (IP address and ports) and storage volumes.

Deployment / StatefulSet: Declaratively defines the desired state of an application. For example, a Deployment might declare, "I need to run 3 replicas of an Nginx Pod, using the nginx:1.14.2 image." K8s' controller continuously monitors the actual state; if it detects that a Pod has failed, it automatically creates a new Pod to replace it, ensuring the replica count always remains 3. StatefulSet is suited to stateful applications (such as databases) that require stable network identity and persistent storage.

Service: Provides a stable, unified access entry point for a group of functionally identical Pods (typically managed by a Deployment). A Service has a virtual IP address (ClusterIP); when traffic is sent to this ClusterIP, K8s automatically load-balances it to a healthy backend Pod. This resolves the problem of Pod IP addresses being dynamic and unstable, enabling service discovery and basic load balancing.

ConfigMap / Secret: Used to decouple application configuration and sensitive information (such as passwords and API keys) from the container image.

PersistentVolume (PV) / PersistentVolumeClaim (PVC): Provide an abstraction for persistent storage for stateful applications. The distributed block storage we discussed in Chapter 10 can be consumed by K8s as PV resources through a CSI (Container Storage Interface) plugin.

Kubernetes' Foundational Capabilities:

1. Application Deployment and Version Management: Through Deployments, applications can be easily deployed, and release strategies such as rolling updates and rollbacks can be achieved.
2. Elastic Scaling: With a single kubectl scale deployment ... --replicas=10 command, a service's instance count can be scaled from 3 to 10. K8s also supports automatic horizontal scaling based on CPU or memory utilization through the Horizontal Pod Autoscaler (HPA).
3. Service Discovery and Load Balancing: Through Services, a microservice can easily discover and invoke another microservice without caring about the specific IP addresses or the number of its backend Pods.
4. Self-healing: K8s automatically detects and replaces unhealthy Pods or nodes, ensuring the overall availability of applications.
5. Resource Scheduling: K8s' scheduler is responsible for placing Pods intelligently and efficiently across the entire cluster, maximizing resource utilization.

Kubernetes provides microservices with a solid "chassis," solving the most fundamental problems of deployment, scaling, and fault tolerance. However, when the number of microservices and the complexity of their interactions grow extremely large, we also need more specialized "traffic management systems": service governance frameworks.

11.1.2 Spring Cloud: A Dedicated Microservices Platform for Java

For teams whose primary technology stack is Java (especially the Spring framework), Spring Cloud is a highly popular and mature suite of microservice governance tools. It is not a brand-new framework but rather a packaging and integration of various excellent microservice components from the industry (many from Netflix's open-source projects), seamlessly integrated with Spring Boot.

Core Components of Spring Cloud

Service Registry and Discovery:

• Components: Eureka, Consul, Nacos.
• Principle: Each microservice instance registers its information (service name, IP, port, etc.) with a service registry (Eureka Server) upon startup. When a service consumer (another microservice) needs to call a service, it first queries the registry for the address list of all healthy instances of that service, then selects one through client-side load balancing. The registry uses a heartbeat mechanism to detect and remove unhealthy instances.

Declarative REST Client and Load Balancing:

• Components: Feign, Ribbon (now replaced by Spring Cloud LoadBalancer).
• Principle: Feign allows developers to call remote REST APIs as if they were local methods, by writing a simple Java interface with annotations. It integrates Ribbon/SC LoadBalancer underneath, which automatically fetches the address list from the service registry and performs client-side load balancing.

API Gateway:

• Components: Zuul (1.x), Spring Cloud Gateway (2.x).
• Principle: The API Gateway is the entry point through which all external requests enter the microservice system. It handles cross-cutting concerns such as request routing, authentication, authorization, rate limiting, and logging, allowing backend business microservices to focus on their own logic.

Circuit Breaker:

• Components: Hystrix, Resilience4j, Sentinel.
• Principle: In a complex distributed system, the latency or failure of a downstream service can propagate up the call chain, eventually triggering a "snowball" collapse of the entire system. The circuit breaker acts like a fuse in an electrical circuit: it monitors calls to a given service. When the failure rate or latency exceeds a certain threshold, the breaker "trips," and for a period of time afterward all calls to that service fail fast, returning a degraded result (such as a cached default value) instead of waiting indefinitely. This protects the caller and gives the callee time to recover.

Distributed Configuration:

• Components: Spring Cloud Config, Nacos, Apollo.
• Principle: Store the configuration files of all microservices centrally in one place, enabling unified configuration management and dynamic refresh.

Advantages of Spring Cloud: For teams on the Java/Spring stack, it offers a gentle learning curve, seamless integration with the existing ecosystem, and a complete set of solutions for microservice governance. Disadvantages: Language binding. It primarily serves Java applications; for a polyglot microservice system comprising Python, Go, Node.js, and other languages, Spring Cloud is not applicable.

11.1.3 Istio: Language-Agnostic, but Architecture-Specific

To resolve the language binding problem of frameworks like Spring Cloud and to align with the trend of Kubernetes becoming the universal platform, a new generation of microservice governance technology called the service mesh emerged, with Istio being its most famous and most powerful representative.

The Core Idea of the Service Mesh: The Sidecar Proxy

Istio's philosophy is to strip all the complex logic related to inter-service communication (such as service discovery, load balancing, circuit breaking, telemetry, and security) out of the application code and sink it into an independent infrastructure layer.

It accomplishes this by automatically injecting a lightweight network proxy, Envoy Proxy, into every business Pod in the Kubernetes cluster. This Envoy proxy "rides alongside" the business container, much like a motorcycle's sidecar, hence the name Sidecar.

From then on, all network traffic of the business containers within a Pod, whether inbound or outbound, is transparently intercepted and routed through this Envoy Sidecar.

Istio's Architecture: Control Plane and Data Plane

Data Plane: Composed of the cluster of Envoy Sidecar proxies injected into all business Pods. This is where actual data traffic is processed.

Control Plane: Composed of a core component called istiod. It processes no business data packets; it is responsible only for: Synchronizing service and endpoint information from the Kubernetes API Server.

Receiving the traffic rules and policies that administrators define through Istio's custom resources (CRDs, such as VirtualService and DestinationRule).

Dynamically and in real time translating this information and these policies into configurations that Envoy proxies can understand.

Distributing the configurations to all Envoy proxies in the data plane.

How Istio Implements Service Governance:

Intelligent Routing and Traffic Management: Administrators can create a VirtualService object to define extremely flexible routing rules. For example:

Send 90% of traffic to version v1 of a service and 10% to version v2, enabling canary releases.

If a request's HTTP header contains user-agent: Android, forward the request to a specific subset of services.

Inject faults or latency for chaos engineering testing.

Observability Out of the Box: Because all traffic flows through Envoy, Envoy automatically generates detailed telemetry data (Metrics, Logs, Traces) for every request. istiod collects this data and can push it to monitoring systems such as Prometheus (metrics), Jaeger (distributed tracing), and Fluentd (logs), without any modification to the business code.

Powerful Security Capabilities: Istio can automatically enable mutual TLS encryption (mTLS) for all communication within the service mesh, and provide fine-grained authorization policies based on service identity (Service Account).

Resilience and Reliability: Administrators can configure connection pools and load-balancing policies through DestinationRule, and retry, timeout, and circuit-breaking policies through VirtualService.

Advantages of Istio:

Language Agnostic: Completely transparent to applications; supports any programming language.

Non-Invasive to Business Code: Service governance logic is fully decoupled from the business code.

Powerful Features: Offers unparalleled traffic control, security, and observability capabilities.

Cloud Native: Deeply integrated with Kubernetes; it represents the future direction of cloud-native microservice governance.

Disadvantages:

Complexity: Istio's architecture and concepts are relatively complex, with a steep learning curve.

Performance Overhead: Introducing Sidecar proxies adds extra network hops and resource consumption; latency-sensitive applications require careful evaluation of the performance impact.

11.1.4 Commercial Microservices Platforms: Options to Balance Various Needs

Beyond open-source solutions, major cloud providers also offer their own managed microservice platforms, such as AWS App Mesh, Google Cloud's Anthos Service Mesh, and Alibaba Cloud's ASM. These are typically based on Istio or similar self-developed technologies, but they provide simpler user interfaces, deeper integration with cloud services, and commercial support, lowering the barrier to adopting a service mesh for enterprises.

For an AI platform, if the team's technology stack is relatively uniform (for example, pure Java), Spring Cloud may be a good choice for a rapid start. But if the platform needs to support multi-language AI model services (such as Python inference services, Java data processing services, and Go API gateways), and if the goal is long-term architectural evolution and strong governance capabilities, then embracing a cloud-native microservices platform centered on Kubernetes and Istio is the more forward-looking strategic choice.

11.2 Middleware Services

If the microservices platform provides the "skeleton" of an application, then middleware services are the "organs" and "blood vessels" that fill it in, supplying indispensable core capabilities such as message passing, data caching, and persistent storage. In an AI platform, the following three categories of middleware are especially important.

11.2.1 Message Middleware

Message middleware, also known as a message queue (MQ), plays a critical role in distributed systems in enabling asynchronous communication, application decoupling, and traffic peak shaving.

The Core Model: Producer and Consumer

Producer: Responsible for generating messages (for example, a user behavior log or a model-training-completed event) and sending them to the message middleware.

Consumer: Subscribes to and pulls messages from the message middleware for processing.

Broker: The message middleware's own service cluster, responsible for receiving, storing, and forwarding messages.

Application Scenarios in an AI Platform:

1. Asynchronous Task Processing:

Scenario: After completing a prediction, an online inference service needs to record logs, update user profiles, and trigger a billing event. If these operations are performed synchronously, they greatly increase the API's response time.

Solution: After completing the prediction, the inference service simply sends a message containing all the relevant information, an "inference completed event," to the MQ and immediately returns the result to the user. Backend services such as logging, user profiling, and billing act as consumers and asynchronously retrieve and process this event from the MQ.

2. Application Decoupling:

Producers and consumers have no direct invocation relationship; they interact only with the MQ. This means either side can be modified, deployed, or scaled independently without affecting the other. For example, you can add a new consumer (such as a fraud detection service) at any time to process the "inference completed event," without making any changes to the original inference service.

3. Traffic Peak Shaving and Buffering:

Scenario: In a recommendation system, the massive volume of user clicks generated in real time needs to be fed into a stream processing system (such as Flink or Spark Streaming) for near-real-time feature updates. The rate at which user behavior is generated fluctuates, and there may be sudden peaks.

Solution: Use the message middleware as a "reservoir" between the frontend system and the backend processing system. Even if a flood of requests arrives at the frontend, they can be written quickly into the MQ and accumulated there, while the backend system consumes data from the MQ steadily at its maximum processing capacity, preventing it from being overwhelmed by burst traffic.

Selecting Mainstream Message Middleware:

RabbitMQ:

Based on the AMQP protocol, feature-rich, and supports multiple messaging patterns (such as Direct, Fanout, and Topic).

Advantages: Mature and stable, with a user-friendly management interface; supports message acknowledgment and transactions, offering high reliability. It is well suited to business scenarios requiring complex routing and reliable message delivery.

Disadvantages: Lower performance and throughput than Kafka; not suitable for ultra-large-scale data pipeline scenarios.

RocketMQ:

An open-source, high-performance, high-reliability distributed message middleware from Alibaba, widely used in e-commerce and other scenarios.

Advantages: High throughput, supports massive message accumulation, feature-rich (such as transactional messages and delayed messages), and benefits from a strong Chinese community.

Kafka:

Initially developed by LinkedIn, now an Apache top-level project. It is not merely a message queue but is designed as a distributed, partitioned, replicated streaming data platform.

Architecture Features:

Topics and Partitions: Messages in Kafka are categorized by topic. Each topic can be divided into multiple partitions. Partitions are the key to Kafka's parallel processing and high throughput.

Ordering Guarantee: Within a single partition, messages are strictly ordered.

Consumer Groups: Multiple consumers can form a consumer group to collectively consume a topic. Kafka guarantees that a partition is consumed by only one consumer in the group at any given time, thereby achieving consumption load balancing.

Advantages: Extreme throughput. Through techniques such as disk sequential writes and zero-copy, Kafka's throughput can easily reach hundreds of thousands or even millions of messages per second. It is ideally suited as the data bus for big data pipelines, log collection, and stream processing.

Disadvantages: Relatively simple functionality; does not support complex message routing, and its reliability guarantees (such as "at-least-once" and "exactly-once") are more complex to implement than RabbitMQ's.

11.2.2 Cache Middleware

Caching plays a crucial role in any high-performance system. Its core purpose is to store hot data in high-speed storage (typically memory) so as to reduce access to slow backend storage (such as databases and disks), thereby lowering latency and increasing throughput.

Application Scenarios in an AI Platform:

1. User/Item Feature Caching:

Scenario: In a recommendation or advertising system, online inference requires real-time access to a large number of user features (age, gender, historical behavior) and item features (category, tags, price). These features are typically stored in databases or data warehouses, where direct queries perform poorly.

Solution: Cache frequently accessed user and item features in a distributed cache (such as Redis). When a request arrives, the inference service first queries the cache; if there is a hit, it returns directly; if not, it fetches from the backend database and writes the result to the cache for future use.

2. Model Parameter or Intermediate Result Caching: Reusable computation results or portions of a model can be cached.
3. Session Management, Distributed Locks, and More: Caches can also be used to implement common functions in distributed systems.

Selecting Mainstream Cache Middleware:

Redis:

The most popular in-memory key-value database today.

Core Features:

Extremely Rich Data Structures: Redis supports not only simple strings but also a range of powerful built-in data structures, including Lists, Hashes, Sets, and Sorted Sets. This allows it to solve problems far more complex than simple caching. For example, Sorted Sets can easily implement leaderboards, and Lists can implement a simple message queue.

High Performance: Based purely on in-memory operations, with a single-threaded model that avoids the overhead of multi-threaded context switching, and combined with I/O multiplexing, its performance is extremely high, with single-machine QPS reaching the 100,000 level.

Persistence: Supports two persistence methods, RDB (snapshot) and AOF (append-only log), ensuring that data is not lost after a restart.

High Availability and Scalability: Supports master-slave replication for high availability, and distributed scaling through Sentinel or the official Redis Cluster solution.

Status: Redis is almost the de facto standard choice for building distributed cache systems.

Memcached:

A purely distributed, in-memory object caching system.

Advantages: Extremely simple architecture and very high performance.

Disadvantages: Supports only simple key-value strings, has no persistence, and is single-purpose. After Redis emerged, its use cases have been largely superseded by Redis.

11.2.3 Databases (Data Middleware)

Although we have caches, most of an application's "full" state data must ultimately be persisted to a database. The choice of database depends on the data model and the application's requirements for consistency and availability.

Relational Databases (RDBMS):

Representatives: MySQL, PostgreSQL, Oracle.

Model: Based on a strict two-dimensional table structure and the SQL language, and ensuring strong data consistency through ACID (Atomicity, Consistency, Isolation, Durability) transactions. Application in an AI Platform:

Stores structured metadata, such as user information, model version information, training task configurations, and billing records.

For business scenarios with modest data volumes but high requirements for transactions and consistency, RDBMS remains the best choice.

Challenge: The scalability of traditional monolithic relational databases is a major concern. Although it can be extended through read-write splitting and database/table sharding, the resulting architecture becomes very complex.

NoSQL Databases (Not Only SQL):

To meet the challenges of massive data and high concurrency in internet applications, various types of NoSQL databases have emerged.

Key-Value Stores: Such as Redis and DynamoDB. Simple models, high performance, and good scalability.

Document Stores: Such as MongoDB. Data is stored in BSON document format, similar to JSON, with a flexible schema, making these databases well suited to semi-structured data.

Column-Family Stores: Such as HBase and Cassandra. Designed for sparse, wide-table storage of large-scale datasets, with high read/write throughput. Well suited to time-series data such as user behavior logs and monitoring metrics.

Graph Databases: Such as Neo4j and JanusGraph. Specifically designed to store and query graph-structured data (nodes, edges, and properties), and ideal for building knowledge graphs, social networks, risk control, and other applications.

NewSQL Databases:

Representatives: TiDB, CockroachDB, Google Spanner.

Goal: To combine the SQL interface and strong consistency transactions of RDBMS with the distributed, highly scalable nature of NoSQL.

They typically adopt a distributed transaction model similar to Percolator, built on KV engines such as RocksDB, and use Paxos or Raft protocols for data replication and high availability.

Trend: NewSQL is an important direction in the database field, offering an ideal solution for relational data scenarios that require elastic scaling.

Choosing Middleware for an AI Platform:

A typical AI platform's backend middleware architecture is usually a combination of the following:

Use Kafka as a unified data bus and stream processing platform, carrying massive streaming data such as logs and user behavior.

Use a Redis cluster as a high-performance cache for hot features and metadata.

Use MySQL or PostgreSQL (possibly in high-availability clusters) to store core, structured business metadata.

Use MongoDB or Elasticsearch to store semi-structured application logs or documents.

Use NewSQL databases such as TiDB for those challenging scenarios that require both transactions and elastic scaling.

11.3 Application Logging Services

In a distributed system composed of hundreds or thousands of microservices and tens of thousands of Pods, when a problem occurs, how do you quickly determine which service's instance, at which node, experienced what issue at what time? The traditional approach of logging into each machine and searching log files with grep has become completely ineffective.

We must build a centralized, searchable, and visualizable application logging service system. The goal of this system is to automatically and reliably collect, aggregate, and store all logs generated by applications in the cluster in a central location, and to provide powerful search, analysis, and alerting capabilities.

The Classic EFK/ELK Architecture

EFK (Elasticsearch, Fluentd, Kibana), or its predecessor ELK (Elasticsearch, Logstash, Kibana), is the most classic and popular open-source solution for building centralized logging systems.

Architecture Components:

1. Log Collection -- Fluentd / Logstash / Filebeat:

Role: The "log porter." It runs as a DaemonSet on every node in the cluster.

Tasks:

It automatically discovers and collects the standard output (stdout) and standard error (stderr) logs of all containers on the node (these logs are typically written by the container runtime to specific directories on the host, such as /var/log/pods/).

It can also be configured to collect logs that applications write to files.

Before sending the logs out, it can parse and enrich them. For example, it can parse a single line of unstructured text log into structured JSON containing fields such as timestamp, level, and message; or it can automatically attach Kubernetes metadata, such as the source Pod's name, namespace, and labels.

Selection: Fluentd and Logstash are powerful and plugin-rich but consume relatively more resources. Filebeat is a lighter-weight collector from Elastic, typically used together with Logstash (Filebeat handles collection and forwards to Logstash for parsing and processing).

2. Log Aggregation and Storage -- Elasticsearch:

Role: The "log database and search engine." Elasticsearch is a distributed, horizontally scalable search engine built on Lucene.

Tasks: It receives structured log data from Fluentd/Logstash, creates inverted indexes for this data, and stores them. The inverted index is the core mechanism enabling fast full-text search. Elasticsearch can be composed of multiple nodes to form a highly available cluster.

3. Log Visualization and Query -- Kibana:

Role: The "log dashboard." Kibana is a web UI that interacts with the Elasticsearch backend.

Tasks: Through Kibana's interface, users can use a powerful query language (KQL or Lucene query syntax) to perform real-time full-text search, filtering, and aggregation over the massive volume of logs stored in Elasticsearch. It can also turn query results into various charts and dashboards, enabling visual analysis of log data.

The EFK Workflow:

1. The application prints logs to standard output in the container.
2. The Fluentd agent on the node collects these logs.
3. Fluentd parses and enriches the logs, adding K8s metadata.
4. Fluentd sends the structured JSON logs in batches to the Elasticsearch cluster.
5. Elasticsearch indexes and stores these logs.
6. Operations or development personnel open a browser and access Kibana.
7. In Kibana, they enter a query condition (for example, kubernetes.pod_name: "my-app-pod-xyz" AND level: "ERROR") to quickly find all relevant error logs.

A Modern Cloud-Native Logging Solution: Loki

Although EFK is powerful, its resource consumption (particularly Elasticsearch's demands on memory and disk) is also considerable. In recent years, the Loki project from Grafana Labs has offered a lighter-weight, lower-cost cloud-native logging solution.

Loki's Core Idea: Index Only Metadata

Loki holds that, for logs, full-text indexing is "over-engineered" and expensive. In most troubleshooting scenarios, we first narrow the scope through metadata (labels) (for example, which application? which Pod? which node?), and only then perform a text search (grep) within the selected small range of log streams.

Therefore, Loki indexes only the labels of logs (such as app="nginx" and pod="my-pod-1"), while the original log line content is compressed and stored in chunks in cheaper object storage (such as S3 or MinIO).

Architecture:

Promtail: Loki's log collection agent, similar to Filebeat, responsible for collecting logs, extracting labels, and sending them to Loki.

Loki: The core service, responsible for receiving logs, indexing labels, and writing log chunks to backend storage.

Grafana: Loki integrates seamlessly with Grafana. Users can query logs in Grafana using its query language LogQL, much as they would query Prometheus metrics, and correlate them with metrics and tracing data in the same dashboard.

Advantages:

Very Low Cost: Because only labels are indexed, the index size and resource consumption are far smaller than those of Elasticsearch. The backend can use inexpensive object storage.

Easy to Operate: The architecture is simpler, and operational costs are lower.

Seamless Integration with the Monitoring Ecosystem: Together with Prometheus and Grafana, it forms the "golden combination" of cloud-native observability.

For an AI platform whose resources are not unlimited and that already uses a Prometheus/Grafana monitoring system, adopting Loki as the logging solution is a highly attractive and cost-effective choice.

11.4 Chapter Summary

In this chapter, we made the critical leap from underlying infrastructure to upper-layer application platform. We are no longer merely "infrastructure engineers" who provide computing power, network, and storage; we have become "city planners" building a thriving, efficient AI application ecosystem. We designed and implemented a modern, cloud-native machine learning application development and runtime platform.

Our construction journey began with the cornerstone of the platform: the microservices platform. We recognized that decomposing large monolithic AI applications into independent microservices is an inevitable choice for improving development efficiency, scalability, and reliability. We delved into Kubernetes, the "operating system" of the cloud-native era, and understood how, through core abstractions such as Pods, Deployments, and Services, it provides microservices with deployment, scaling, self-healing, and basic service discovery capabilities. On this foundation, we compared two mainstream service governance approaches: the Spring Cloud suite tailored to the Java ecosystem, and the language-agnostic, powerful, future-oriented service mesh technology Istio.

Next, we built critical public service facilities for this "microservice city": middleware services.

We learned how message middleware, represented by Kafka, through its powerful asynchronous decoupling and traffic peak shaving capabilities, becomes the "data bus" that connects microservices and builds real-time data pipelines.

We analyzed cache middleware, centered on Redis, and understood how, by placing hot data (such as user features) into memory, it dramatically reduces application latency and improves system performance.

We also surveyed database selection, from traditional relational databases like MySQL, to NoSQL for handling massive data, to NewSQL (such as TiDB) that combines the strengths of both, matching appropriate solutions to the different data persistence needs of applications.

Finally, we installed the "black box" and "surveillance cameras" for this complex distributed system: application logging services. We learned how the classic EFK (Elasticsearch, Fluentd, Kibana) architecture achieves centralized log collection, storage, indexing, and visual querying. At the same time, we understood how Loki, a lighter-weight newcomer in cloud-native logging, through the philosophy of "indexing only metadata," greatly reduces storage and operational costs while preserving core functionality.

Through the practice in this chapter, we have built a functionally complete, technologically advanced AI PaaS platform. Data scientists and algorithm engineers can now conveniently develop and debug on this platform and package their models into highly available, scalable microservices. The completion of this platform marks the evolution of our AI infrastructure from a primitive "compute farm" into a modern "AI factory" capable of continuously generating value. In the chapters that follow, we will explore how to conduct efficient operations, management, and cost control for this already-built "AI city."