祝融说。

Prologue: The Million Missing Points Mystery

The "Starfish E-commerce" platform has a popular member points system. (This case is a synthetic example; its numbers are order-of-magnitude illustrations.) Every time a user successfully places an order, the order system calls the points system's API to add the corresponding points to the user's account. This flow had been running stably for several years.

However, during a quarterly financial audit, a data analyst discovered a shocking discrepancy. The total points that should have been issued, calculated from all "Completed" orders in the order system's t_order table, was nearly one million points less than the total points actually issued, as recorded in the points system's t_points_transaction table.

This discovery put the entire technology department on edge. A million points might not be a large monetary amount, but it meant that somewhere in an unnoticed corner of the system, user assets were being silently "consumed." This was a more frightening problem than a system crash -- silent data inconsistency.

The team immediately formed a special task force to investigate. They faced enormous challenges:

1. Hard to reproduce: The discrepancy was the result of months of accumulation, involving tens of millions of order and points records. What specific scenarios triggered the issue? Was it network jitter? A lost message in the message queue? A bug in a specific version of the code? No one knew.
2. Logs had expired: Detailed application logs were typically only retained for 7 to 30 days and could no longer cover the full problem period.
3. Lack of reconciliation mechanism: No one had ever thought to compare the general ledgers of these two systems. Everyone assumed that as long as the API call succeeded, the data was consistent.

After weeks of painful "data archaeology" -- writing complex scripts to compare the massive amounts of data from the two systems day by day -- the team finally identified several causes:

• Cause One: During a few minutes of a database master-slave switchover, the order system's transaction for writing the "Completed" status succeeded, but the subsequent "Add Points" message sent to the message queue was lost due to a connection pool exception.
• Cause Two: The points system had once been deployed with a buggy version. For certain orders containing virtual products, it would erroneously consider the points as having been issued without actually writing to the database. The bug was fixed a week later, but it had already created a data gap.
• Cause Three: A background script used for manual points compensation, due to not accounting for concurrency, would exit prematurely in some cases, leaving some users' points un-added.

After identifying the causes, the team spent several more weeks developing, testing, and executing a one-time, large-scale data repair script before finally filling the gap of one million points.

This "Million Points Disappearance Case" taught the team a hard lesson. They realized that in a complex distributed system, even if every individual service is fully tested, and even if every API call has retry and exception handling, the eventual consistency of data over long periods and at large scale is still a matter of probability, not certainty.

Code can have bugs, networks can be interrupted, servers can crash, and people can make mistakes. Relying on "perfect" code to "prevent" all inconsistencies from occurring is an unrealistic fantasy.

We need a more powerful, systematic guarantee mechanism. This mechanism is "reconciliation." And this chapter will argue that reconciliation should no longer be regarded as a post-hoc, passive, manual "error-checking" tool. It must be elevated to the core level of architectural design, becoming an active, automated "immune system" that guarantees the ultimate correctness of the system.

Section 1: Not Just Error-Checking, But Architecture -- Making "Reconciliation" Core Business Logic Instead of an Operations Script

In the traditional view, reconciliation usually looks like this: at the end of each month, a finance person exports the bank statements and the company's internal ledger, uses Excel's VLOOKUP function to compare them line by line, finds discrepancies, generates a discrepancy report, and sends it to the technology department to "fix the data."

This "operations script" style of reconciliation has three fatal flaws:

1. Long cycle, high latency: Usually done in units of days, weeks, or even months. By the time a problem is discovered, the loss may have already become significant, and the golden window for investigation (such as logs, monitoring snapshots) has long passed.
2. Passive response, lacks root cause resolution: It can only discover the "result" inconsistency, but often cannot trace the "cause." The fix is usually a one-time data UPDATE, while the systemic defect that caused the problem may still be lurking.
3. Disconnect between technology and business: It's seen as "dirty work," handled by the operations or data team on a part-time basis, and not integrated into the core product functionality and architecture design.

The "Reconciliation Architecture Theory" we propose aims to completely overturn this mindset.

The core idea of the Reconciliation Architecture Theory is: design the verification and repair of data consistency between systems as a built-in, normalized, automatically running core business logic. The system must not only be responsible for executing business functions, but also continuously prove the correctness of its own execution results.

This is like a modern spaceship: it has not only powerful main thrusters (business logic), but also countless attitude control thrusters (reconciliation logic) that continuously perform tiny attitude adjustments during flight, ensuring the spaceship always stays on the correct trajectory.

To implement this architecture, we need to introduce three core reconciliation patterns in our design.

Reconciliation Pattern One: Patrol-Style Reconciliation

This is the most basic and universal reconciliation pattern. It uses an independent, background reconciliation service to periodically and proactively pull and compare data from two related systems.

Applicable scenarios:

• Both systems have large data volumes, making real-time comparison impractical.
• The business has a relatively high tolerance for data inconsistency, accepting minute-level or hourly-level delays.
• The order and points example is very suitable for this pattern.

Design points:

1. Establish an independent reconciliation service: Reconciliation logic should not intrude into the core code of the order or points services. It should be an independent, horizontally scalable service specifically responsible for performing reconciliation tasks.
2. Define clear reconciliation "slices": Comparing all data from two large systems is impractical. Reconciliation tasks must be broken down into manageable "slices" by time (e.g., compare data from the past 10 minutes every 5 minutes), by user shard, or by geographic region.
3. Efficient data fetching: The reconciliation service needs to call dedicated interfaces provided by the order and points services to efficiently obtain the required data for reconciliation. These interfaces should support paginated queries by time range and slice ID, and only return necessary fields (e.g., order_id, completion_time, expected_points).
4. Comparison and discrepancy recording: The reconciliation service compares the two data sources in memory or using temporary storage. Once a discrepancy is found (e.g., Order A should have earned 100 points, but the points system shows no record or only 90 points), this discrepancy is persisted in a "discrepancy ledger."
5. Automated repair and human intervention:
   • For certain pattern-clear discrepancies (e.g., "order exists, no points record"), an automated repair flow can be triggered (calling the points system's compensation API).
   • For complex discrepancies that cannot be automatically repaired, the discrepancy ledger automatically generates an alert or creates a pending ticket to notify human intervention.

Patrol-style reconciliation architecture diagram:

                  +----------------------+
                  | Reconciler Service   |
                  +----------------------+
                         | (1) Fetch Slice
           +-------------+-------------+
           |                           |
           v                           v
+------------------+        +-------------------+
|  Order Service   |        |   Points Service  |
| (Source of Truth |        | (Target System)   |
| for "Intent")    |        |                   |
+------------------+        +-------------------+
    ^  (4a) Auto-Fix         | (2) Compare Data
    |                        |
    |                        v
    |              +----------------------+
    +--------------| Discrepancy Ledger   |
                   +----------------------+
                         | (4b) Manual Alert
                         v
                   +----------------------+
                   |   Monitoring/Alert   |
                   +----------------------+

With this pattern, the "Million Points Disappearance Case" is more likely to be detected before losses keep growing, but it cannot be promised never to happen. Reconciliation can itself miss or create errors through a shared bad data source, omitted slices, failed monitoring, or faulty compensation logic. After a database failover loses messages, the next slice detects and compensates the missing points only if its inputs are independent, it covers the affected orders, and the job runs on time. The reconciliation job therefore also needs monitoring, sample checks, and a human fallback.

Patrol-style reconciliation is the last and most solid line of defense against unknown bugs and "black swan" events.

Reconciliation Pattern Two: Ticket-Style Reconciliation

While patrol-style reconciliation is powerful, it has relatively high latency. For near-real-time, transactional scenarios (such as payment, order placement), we need a lighter, more timely reconciliation method. This is the "ticket-style reconciliation" we have already glimpsed in the "Double-Ledger Pattern" from the previous chapter.

Applicable scenarios:

• Involving a one-time, critical business interaction between two systems.
• The business has a very low tolerance for inconsistency, requiring prompt detection and repair.
• Interaction between the payment system and the order system.

Design points: The core of "ticket-style reconciliation" is delegating the "responsibility for reconciliation" to the caller. The caller's (order system) "intent ledger" (t_payment_request) itself is a natural, fine-grained reconciliation checklist.

1. Clear ticket state machine: Records in the intent ledger must have a clear state machine, at least including:
   • PENDING: Intent created, not yet sent.
   • SENT: Request sent, awaiting final result.
   • CONFIRMED: Clear success result received from the other party.
   • FAILED: Clear failure result received from the other party, or maximum retry count reached.
2. Timeout equals anomaly: Any "ticket" in SENT status for longer than a certain time (e.g., 5 minutes) should be considered anomalous.
3. Proactive polling query: The order system must have a background task specifically to poll those timed-out SENT tickets and proactively call the payment system's query API (getPaymentStatusByRequestId) to get their final status.
4. Idempotency guarantee: Both the payment system's execution API and query API must guarantee idempotency based on the ticket ID (request_id).

Ticket-style reconciliation breaks down the reconciliation behavior from a global, batch process into countless microscopic, transaction-level self-verification processes. It gives every cross-service interaction a built-in "timeout-query-calibration" closed-loop mechanism, greatly enhancing the system's real-time error-correction capability.

Reconciliation Pattern Three: Stream-Style Reconciliation

For systems with massive data volumes and extremely frequent changes (such as logs, real-time transactions), batch patrols and point-in-time ticket queries may both be overwhelmed. In such cases, we need a more modern, stream-processing-based reconciliation method.

Applicable scenarios:

• High-throughput data synchronization scenarios.
• For example, a business database needs to synchronize changes to a data warehouse or search engine in real-time.

Design points:

1. Capture change data stream (CDC): All data changes (Insert, Update, Delete) from the source system (e.g., business database) are captured via CDC technology (e.g., Debezium), forming an immutable event stream that is pushed to a message queue (e.g., Kafka).
2. Dual-stream Join: The target system (e.g., data warehouse) also forms an event stream from the data changes it receives.
3. Real-time reconciliation engine: A stream processing engine (e.g., Flink, ksqlDB) subscribes to both event streams and performs a real-time Join based on a common key (e.g., order ID) and a time window.
4. Real-time discrepancy detection: If, within a time window, a certain key appears only in the source stream but not in the target stream, a "data loss" discrepancy is detected. And vice versa.

Stream-style reconciliation reduces reconciliation latency from minutes to seconds or even milliseconds. It brings the batch reconciliation paradigm into the realm of real-time computing, making it possible to guarantee data consistency in scenarios that demand high timeliness.

In summary: making reconciliation a core part of the architecture means we no longer view the system as an ideal, deterministic state machine. We acknowledge its fragility and fallibility, and embed within it a powerful "immune system" composed of three patterns -- patrol, ticket, and stream -- each with different granularity and timeliness. This immune system evolves our distributed system from a fragile, "sickly" constitution into a robust "organism" capable of continuous self-healing.

Section 2: Fault-Tolerant Design -- When Code Can't Be Trusted, Use Data Redundancy for "Disaster Replay" and Repair

Reconciliation systems can detect and repair "result" inconsistencies. But sometimes, the root cause lies in a deviation in the business logic itself, leading to a "process" error. For example, a bug causes incorrect calculation of an order's discount amount. Even though the order and payment amounts can be reconciled, the amount itself is wrong.

In this situation, we need a deeper level of fault tolerance: "Disaster Replay."

The idea of "Disaster Replay" is: if our code logic cannot be trusted, then the only things we can trust are the most primitive, unprocessed input data received by the system. We must be able, after fixing the bug, to use these raw inputs to "replay" the business process and generate the correct output results.

To achieve this, the key is to implement thorough, immutable data redundancy for "requests" and "events."

Principle One: Persist Every "Command"

In the CQRS (Command Query Responsibility Segregation) pattern, a "Command" is a request intended to change the system's state. For example, "create an order," "modify a product price," "add points to a user."

Typically, our Controller receives an HTTP request, parses it into a DTO (Data Transfer Object), and passes it in memory to the Service layer for processing. Once processing is complete, this original request DTO is lost.

Fault-tolerant design requires us to: before executing any business logic, first serialize and persist this complete, original "Command" object.

t_incoming_command_log table:

This table is our system's "black box." It records all external inputs that attempt to change the system's state.

The power of Disaster Replay:

• Scenario: We discover that the processing logic for CreateOrderCommand has a serious bug. It calculates shipping fees incorrectly for certain special regions. This bug has been running in production for two days, causing shipping fee errors on thousands of orders.
• Fix process:
  1. Stop accepting new commands (or enter read-only mode).
  2. Fix the code bug and deploy the new version of the service.
  3. Locate affected commands: SELECT * FROM t_incoming_command_log WHERE command_type = 'CreateOrderCommand' AND received_at BETWEEN [bug_start_time] AND [bug_end_time].
  4. Write a "replay" script: This script reads the payload of these selected commands and re-executes the order creation process using the new version of the code logic (usually in an isolated environment, or through a special "correction mode").
  5. Generate correction data: The replay calculates the correct shipping fee and generates data correction scripts (UPDATE t_order SET shipping_fee = ? WHERE id = ?), or generates refund tickets for the difference.
  6. Execute the correction and restore service.

Without this "command log," fixing this problem would be a nightmare. We would have to guess what the input for each order was at the time, making the fix process full of uncertainty. With it, we have a deterministic, replayable input source, making precise, large-scale fixes possible.

Principle Two: Record Every "Domain Event"

"Commands" are external inputs, while "Domain Events" are the results of internal state changes within the system. For example, "Order Created," "Payment Succeeded," "Points Added."

In event-driven architectures, these events are typically consumed after being published to a message queue and then discarded. But fault-tolerant design requires us to also persist these key domain events.

t_domain_event_log table:

This table is the "historical film strip" of the system's internal state changes. It records every state change of every key business object (aggregate root).

Combining the command log with the event log, we can achieve ultimate fault tolerance:

• Scenario: We not only fix the shipping fee calculation bug, but also want to know the impact of this bug on the downstream "shipment notification" process.
• Replay and analysis:
  1. We can replay the CreateOrderCommand to generate the correct OrderCreated event.
  2. Then, we can compare (diff) the "old, incorrect OrderCreated event" with the "new, correct OrderCreated event."
  3. Through this diff, we can precisely analyze: which orders had their amounts changed, which orders had their shipping address information incorrectly processed at the time, etc.
  4. Based on this precise analysis, we can decide whether to send an "Order Information Correction" notification to the downstream shipping system, or whether to cancel wrong notifications that were already sent.

When code cannot be trusted, the only thing we can do is fall back to the most trustworthy data source. The t_incoming_command_log and t_domain_event_log are the two "ledgers" established through data redundancy -- they are the "immutable anchors" we can rely on in the "chaos" of code logic. They are the prerequisites for implementing "Disaster Replay" and precise repairs, and the foundation for building a truly "anti-fragile" system.

Chapter Summary: From "Builder" to "City Maintainer"

In this chapter, we explored the ultimate guarantee for the long-term stability of distributed systems -- the Theory of System Reconciliation. We completely overturned the traditional notion of "reconciliation = operations script," elevating it to a core architectural design philosophy.

We first demonstrated why reconciliation must be treated as core business logic. In a distributed world where errors are inevitable, continuous self-verification and repair are the only way for a system to maintain correctness. To this end, we introduced three reconciliation patterns with different granularities:

1. Patrol-style reconciliation: Suitable for large-volume, high-latency scenarios, serving as the last line of defense for system eventual consistency.
2. Ticket-style reconciliation: Suitable for transactional, low-latency scenarios, delegating reconciliation responsibility down to each interaction.
3. Stream-style reconciliation: Suitable for high-throughput, real-time scenarios, bringing reconciliation into the era of stream computing.

Next, we explored how to design for higher-level fault tolerance when the code logic itself is wrong. We introduced the idea of "Disaster Replay" and emphasized that its implementation depends on the redundancy and persistence of key data:

1. Persist every "Command": By establishing a "command log," we retain a replayable "black box" for the system, making precise repairs possible.
2. Record every "Domain Event": By establishing an "event log," we take a "historical film strip" of the system's state changes, providing a basis for in-depth analysis and downstream compensation.

Having completed this chapter, our way of thinking should undergo a profound shift: we are not just "builders" of software, but "long-term maintainers" of the digital city we have constructed.

An excellent city planner, when designing a water supply system, not only designs the supply pipes (business logic), but also simultaneously designs a leak detection system (reconciliation logic) and emergency repair plans (Disaster Replay). They know that pipes will eventually age and leak. Their goal is not to build a pipe that never leaks, but to build a resilient water supply system that can quickly detect a leak, locate the problem, and repair it with minimal cost.

This is the essence of the "Theory of System Reconciliation." It requires us to view the systems we create with a more humble and realistic eye. Acknowledge their imperfections, and for those imperfections, design elegant, automated "repair mechanisms" in advance.

At this point, we have completed our full decompression journey from the micro, through the meso, to the macro level. In the final part, the "Action Guide," we will translate all these theories and patterns into an executable roadmap, telling you how to identify problems in your own legacy systems step by step, and safely implement refactoring.