祝融说。

"Good fences make good neighbors." — Robert Frost, "Mending Wall"

If mesoscopic decompression is about planning clear functional zones (table structures) within a single city (a monolithic system), then macroscopic decompression is about drawing national borders between different cities (microservices) and establishing the diplomatic and trade agreements between them.

In a monolithic application, data consistency is guaranteed by the database's ACID transactions. But in a microservices architecture, a complete business process -- such as "a user placing an order" -- may span multiple independent systems like the order service, inventory service, payment service, and user service. Each system has its own database, and traditional database transactions are ineffective here.

This brings us to the most central and thorny problem in distributed systems: how do we ensure the ultimate correctness of a business process across an unreliable network and potentially faulty services?

Many teams attempt to solve this problem through complex distributed transaction protocols (such as two-phase commit 2PC, TCC, Saga). While theoretically feasible, these approaches often introduce immense implementation complexity and performance overhead in practice, and can even lead to system-wide "deadlock" disasters.

This part will propose a completely different, more resilient approach. Instead of pursuing the "atomicity" of cross-system transactions, we will embrace the reality of "eventual consistency" in a distributed world. We will learn how to use "strategic data redundancy" and design that treats "reconciliation as core business logic" to build an "anti-fragile" system that can self-heal and eventually reach correctness, even in the face of partial service failures and network partitions.

This is a shift in architectural thinking: from trying to "prevent" all errors from occurring, to designing a system that can "recover" from errors.

Prologue: The "Rashomon" of the Payment System and Order System

On the "Starfish E-commerce" platform, the order system and payment system are two core microservices maintained by different teams. A typical payment flow is as follows:

1. The user clicks "Pay" in the order system.
2. The order system generates a payment request and calls the payment system's API, passing information such as the order ID and amount.
3. The payment system processes the payment and synchronously returns a processing result (success/failure) to the order system.
4. The order system updates the order status to "Paid" or "Payment Failed" based on the returned result.

One afternoon, an angry user complained: "I clearly received a debit SMS from my bank, so why does your app still show my order status as 'Pending Payment'?"

Engineer Xiao Liu, responsible for troubleshooting in the order system, and Xiao Zhang, responsible for the payment system, fell into a classic distributed system "Rashomon":

• Xiao Liu (Order System): He checked the order system's logs and found that when the payment system's API was called, after a 30-second wait, it returned a "Network Timeout" exception. According to the code logic, a timeout is treated as a payment failure, so the order status was not modified. Xiao Liu insisted: "The payment system did not give me a clear success response. The responsibility lies with you."
• Xiao Zhang (Payment System): He checked the payment system's database and found a payment record for this order with a status of "Success." He explained: "My system did successfully process this payment and has already paid the money to the bank. It's possible that due to network jitter, the connection broke when I was sending the success result back to you. The money has been deducted. It's your order system that didn't handle the final status correctly."

Who is right? They are both right.

From within their respective systems, each system's behavior is consistent with its local logic. The order system is correct to keep the order as "Pending Payment" because it received no success response. The payment system is correct to record the transaction as "Success" because it successfully processed the payment.

The problem lies at the boundary between the two systems. This boundary -- the API call that timed out -- became a black hole of information. The order system doesn't know if the payment system received the request. The payment system doesn't know if the order system received its success response.

This "Rashomon" reveals a harsh truth about distributed systems: You can never fully trust another service. You cannot trust its availability (it might be down), you cannot trust its response (it might time out or return an error), and you cannot even trust its understanding of "the facts" (it might, due to a bug or data inconsistency, tell you an incorrect state).

So, in such a "dark forest" of distrust, how do we build reliable business processes? The answer lies not in more complex RPC frameworks or longer timeouts, but in changing our fundamental view of "data" and "boundaries." We need to acknowledge the existence and unreliability of boundaries, and bridge the trust gap by retaining data copies on both sides of the boundary.

Section 1: The "Double-Ledger Pattern" -- Separating Request Records from Execution Records

To solve the "Rashomon" problem, we need to introduce a powerful macroscopic decompression pattern: the "Double-Ledger Pattern."

The idea for this pattern originates from the double-entry bookkeeping method in accounting. In double-entry bookkeeping, every transaction is recorded in at least two different accounts (one debit, one credit), and the total amounts on the debit and credit sides must be equal. This mechanism inherently provides a verification and reconciliation capability.

In distributed systems, we can borrow this idea. When one service (the caller, such as the order system) requests that another service (the callee, such as the payment system) perform a critical operation, we should not rely solely on that instantaneous, unreliable API call. Instead, we should establish independent, persistent "ledgers" inside both systems to record two different aspects of this interaction.

1. Caller's Ledger: The Intent Ledger

   • This ledger records the clear "intent" of what the caller "wants to do."
   • In the order system, this corresponds to a t_payment_request table.

2. Callee's Ledger: The Execution Ledger

   • This ledger records the final result of what the callee "actually did."
   • In the payment system, this corresponds to the t_payment_transaction table (this table usually already exists).

Let's see how to apply this pattern to refactor the fragile payment flow.

Refactoring the Payment Flow: Introducing the "Intent Ledger"

Old flow (fragile): Order System Controller -> Order System Service (direct RPC call) -> Payment System

New flow (robust):

Step One: The order system records the "payment intent" When the user clicks "Pay," the order system does not make the RPC call directly. Its first step is to create a "payment intent" record in its own database.

t_payment_request table (in the order system's database):

Order system PaymentService (new logic):

public PaymentRequest initiatePayment(Order order) {
    // 1. Check if the order is payable
    // ...
  
    // 2. (Core) Create and persist the payment intent
    PaymentRequest request = new PaymentRequest(
        order.getId(),
        order.getAmount(),
        RequestStatus.PENDING
    );
    paymentRequestRepository.save(request);
  
    // 3. Return this request object, or just its ID
    return request;
}

This step is crucial. By persisting the "payment intent," the order system creates a "task list" for itself. Even if the system crashes before making the RPC call, as long as this PENDING request record exists, the system can recover and continue this process upon restart.

Step Two: Send the request asynchronously The order system now has an independent background task (could be a scheduled task or a message queue consumer) specifically responsible for scanning this "intent ledger" and sending requests to the payment system.

PaymentRequestProcessor (in the order system):

@Scheduled(fixedRate = 60000) // Execute every minute
public void processPendingRequests() {
    List<PaymentRequest> pendingRequests = paymentRequestRepository.findPending();
    for (PaymentRequest request : pendingRequests) {
        try {
            // Update status to SENT, and record the send time
            request.setStatus(RequestStatus.SENT);
            request.setLastSentAt(new Date());
            request.incrementRetryCount();
            paymentRequestRepository.save(request);
          
            // Make the RPC call (IMPORTANT: must be idempotent)
            PaymentResult result = paymentServiceClient.createPayment(request);
          
            // (Success path) Based on the clear success response, confirm the intent
            if (result.isSuccess()) {
                confirmPaymentRequest(request, result);
            }
            // If the payment system explicitly returns failure, mark the intent as failed
            else {
                 failPaymentRequest(request, result.getErrorMessage());
            }
          
        } catch (TimeoutException e) {
            // (Timeout path) Do nothing!
            // Keep the request in SENT status, waiting for the next poll or reconciliation
            log.warn("Payment request {} timed out. Will retry later.", request.getId());
        } catch (Exception e) {
            // Other unknown exceptions, also keep SENT status
            log.error("Failed to process payment request {}.", request.getId(), e);
        }
    }
}

Step Three: The payment system records the "execution result" When the payment system receives the request, its t_payment_transaction table (execution ledger) plays the role of recording the final fact. To prevent duplicate payments, the payment system's API must be idempotent. Typically, we can use the request_id passed by the order system as the idempotency key.

Payment system PaymentController (pseudo-code):

public PaymentResult createPayment(PaymentCreationRequest dto) {
    // 1. Check idempotency: has this request_id already been processed?
    Transaction existingTx = transactionRepo.findByRequestId(dto.getRequestId());
    if (existingTx != null) {
        // If already processed, directly return the result from that time
        return PaymentResult.fromTransaction(existingTx);
    }
  
    // 2. Execute payment business logic
    // ...
  
    // 3. (Core) Create and persist the execution result
    Transaction newTx = new Transaction(
        dto.getRequestId(),
        dto.getOrderId(),
        // ...
        TransactionStatus.SUCCESS // or FAILED
    );
    transactionRepo.save(newTx);
  
    // 4. Return the result
    return PaymentResult.fromTransaction(newTx);
}

Step Four: Status confirmation and reconciliation Now, we have two independent ledgers. The eventual consistency of the business process depends on how these two ledgers are synchronized.

• Proactive Confirmation: In PaymentRequestProcessor, when we receive a clear response from the payment system, we update the status of t_payment_request to CONFIRMED or FAILED.
• Passive Reconciliation: For requests stuck in SENT status due to timeouts, we need a reconciliation mechanism. The order system could provide an interface for the payment system to proactively notify the order system when a payment status changes definitively (e.g., through an asynchronous callback). Alternatively, the order system could have a higher-latency reconciliation task that periodically queries the payment system for the final status of those SENT requests.

The End of the Rashomon

Let's see how the "Double-Ledger Pattern" solves the "Rashomon" problem from the prologue.

• Scenario: The order system timed out after sending the request.
• Order system state: In the t_payment_request table, there is a record request_id=R-123, status=SENT. The order table t_order's status is still Pending Payment.
• Payment system state: In the t_payment_transaction table, there is a record request_id=R-123, status=SUCCESS.

Now, even though the user sees an inconsistent interface, our system's data layer is self-consistent and recoverable.

When the order system's reconciliation task runs, it will find this SENT request. It will take request_id=R-123 and call a query interface on the payment system: getPaymentStatusByRequestId('R-123').

The payment system will look up its "execution ledger" and clearly tell the order system: "This request has already succeeded."

Upon receiving this definitive answer, the order system's reconciliation task will:

1. Update the status of R-123 in the t_payment_request table to CONFIRMED.
2. Trigger business logic to update the corresponding order's status in the t_order table to Paid.

The problem is automatically and deterministically resolved. No more endless arguments between Xiao Liu and Xiao Zhang, and no more manual data corrections.

The essence of the "Double-Ledger Pattern" is introducing a persistent "intent ledger" on the caller's side, transforming an unreliable, synchronous RPC call into a reliable, eventually consistent asynchronous flow. It shifts the system's focus from "how to guarantee the success of a single communication" to "how to guarantee the eventual consistency of two ledgers." The latter is a much more manageable problem.

Section 2: Why Must Microservices Retain Data Copies Between Them?

The core of the "Double-Ledger Pattern" is creating a t_payment_request table in the order system. This table is essentially a "partial copy" and "pre-state machine" of the payment system's t_payment_transaction table.

This raises a highly controversial yet crucial topic in microservice design: should we redundantly store data between services?

Traditional microservice theory, especially those heavily influenced by Domain-Driven Design (DDD), often emphasizes the independence of "bounded contexts" and the "single ownership" of data. It tells us that all information about "payment" should have its sole, authoritative source in the payment service. The order service should not "know" any internal details about payments, let alone copy its data.

This principle is correct in an ideal world. It ensures data consistency and model purity. However, in a real, unreliable distributed environment, dogmatic adherence to this principle often leads to a dramatic decrease in system availability and resilience.

The Triple Benefits of Data Redundancy

Strategically retaining data copies at service boundaries brings at least three decisive benefits:

1. Improved Availability and Performance

• Scenario: The order system needs to display an order list, and each order needs to show its payment status (Pending Payment, Processing Payment, Paid).
• Non-redundant design: Every time the order list is rendered, the order system must iterate through each order in the list and make a real-time, synchronous call to the payment system's getPaymentStatusByOrderId() API.
  • Consequences: 1. Performance Disaster: A single list query could trigger dozens of cross-service RPC calls (N+1 problem). 2. Availability Coupling: If the payment system experiences jitter or goes down, the entire order list page will fail to load or load extremely slowly. The order system's core functionality is "held hostage" by the payment system's availability.
• Redundant design: The order system redundantly stores a payment_status field in its own t_order table. This field's state is kept eventually consistent by listening to domain events published by the payment system (e.g., PaymentCompletedEvent).
  • Consequences: 1. High Performance: Rendering the order list requires only a single local database query. 2. High Availability: Even if the payment system goes down, the order list can still be displayed normally (at most, the payment status is a few seconds behind). The order system and payment system achieve "read de-coupling."

2. Enhanced Business Process Resilience

This is the value demonstrated by the "Double-Ledger Pattern." The t_payment_request data copy allows the payment flow to tolerate temporary unavailability of the payment system and network timeouts.

• Non-redundant design: If the payment system goes down while the order system is calling it, the order system only gets an exception. The "intent" of this payment is lost. The user must retry manually, creating a poor experience.
• Redundant design: If the payment system goes down, the order system simply cannot send the PENDING request. It waits quietly. When the payment system recovers, the background task automatically retries, and the entire flow automatically recovers, potentially without the user even noticing.

This "intent" copy acts as a "circuit breaker" and "retrier" within the business process. It isolates the failure domains of the two systems, preventing the failure of one system from cascading like dominoes and collapsing the entire business chain.

3. Preserving Historical Context Snapshots

Data redundancy between services has another often-overlooked huge benefit: it "freezes" the context at the time of the interaction.

• Scenario: The order system passed the amount amount=100 to the payment system. Payment was successful. One month later, due to an operations campaign adjustment, the order system, due to a bug, incorrectly modified this order's amount field to 90.
• Non-redundant design: When the finance department performs monthly reconciliation, they will find a discrepancy: the order system shows a receivable of 90, while the payment system records an actual receipt of 100. Investigating this issue would be very difficult because the "original evidence" in the order system has been overwritten.
• Redundant design: In the order system's t_payment_request table, we recorded amount=100. This table's data is immutable. It clearly snapshots "at the moment the payment was initiated, the order system believed the amount payable was this much."
  • Consequences: When a discrepancy appears during reconciliation, we have two immutable, comparable data sources: the order system's t_payment_request.amount and the payment system's t_payment_transaction.amount. We can easily identify that a subsequent change to the order data caused the inconsistency, rather than the payment flow itself having a problem.

This data copy becomes a "time capsule" protecting against future data corruption.

The Cost and Principles of Redundancy

Of course, data redundancy is not without cost. Its main cost is "consistency." We must accept that the payment_status copy in the order system and the authoritative payment status in the payment system might be inconsistent for a short period.

Therefore, when deciding whether and how to redundantly store data, we must follow several key principles:

1. Clearly define data ownership: It must be clear which service is the "authoritative source" for the data. For payment status, the payment system is the authority.
2. Accept eventual consistency: For redundant data, the business must tolerate some latency and inconsistency.
3. Choose an appropriate synchronization mechanism: Data can be synchronized through asynchronous messaging (event-driven) or periodic reconciliation (batch processing). Avoid synchronous dual-writes, as they re-introduce availability coupling.
4. Only redundantly store necessary data: Don't copy the payment system's entire table. Only redundant those fields that are critical for improving availability or resilience at the boundary interaction (such as ID, status, amount).

The conclusion is: in a microservices architecture, to gain system availability, resilience, and auditability, strategic data redundancy that acknowledges eventual consistency is not just acceptable, it is necessary. The pure, non-redundant "single ownership" model is a "lab architecture" that cannot survive in the real world.

Chapter Summary: Establish Your Own "Customs Office" at the Boundary

In this chapter, we entered the macroscopic distributed world and confronted the thorniest problem in microservices architecture: "trust."

Starting from the "Rashomon" case of the order system and payment system, we revealed the fragility of relying on instantaneous RPC calls. We pointed out that at an unreliable network boundary, neither party can fully trust the other, nor can either know the final result of a single communication.

To establish certainty at an untrusted boundary, we introduced the "Double-Ledger Pattern." The core of this pattern is establishing a persistent "intent ledger" on the caller's side (e.g., t_payment_request) to record "what you want to do," while the callee's "execution ledger" (e.g., t_payment_transaction) records "what was actually done." Through asynchronous polling and periodic reconciliation, the eventual consistency of these two ledgers is ensured. This pattern transforms a fragile synchronous call into a robust, recoverable asynchronous flow.

Next, we delved into the design philosophy behind this pattern -- microservices must retain data copies between each other. We challenged the dogmatic "single data ownership" principle and argued that strategic data redundancy brings tremendous benefits in three areas:

1. Improved availability and performance: Through read de-coupling, preventing core functions from being held hostage by downstream services.
2. Enhanced business process resilience: Through "intent" copies, enabling fault isolation and automatic retries.
3. Preserving historical context snapshots: Through "time capsules," protecting against future data corruption.

Ultimately, we can draw a vivid conclusion: designing a microservice is like managing a country's border. You cannot naively assume your neighbor will always be friendly and the roads will always be clear. You must establish your own "customs office" (intent ledger), carefully inspecting and recording every "entry and exit" request. This "customs" record is your own, independent version of the facts about boundary interactions. It allows you to maintain stable internal operations even when disconnected from the outside world, and once the connection is restored, to methodically engage in "reconciliation" with your neighbor and ultimately reach a consensus.

We have learned how to establish trust at the boundary. But in a complex distributed system, even if every boundary is robust enough, the long-term accumulation of tiny errors can still cause the overall system state to drift. How do we detect and repair such drift? That is the question the next chapter -- "The Theory of System Reconciliation" -- will answer. We will elevate "reconciliation" from a passive, manual error-correction activity in the financial domain to an active, automated core design principle at the architectural level.