祝融说。

"It works on my machine." "I tested it, it should be fine." "I think this change is simple, it won't affect anything else."

In software development, these judgments based on "feelings" and "should" are the most common breeding grounds for production incidents. Human intuition is extremely unreliable when facing a complex system composed of millions of lines of code, countless interdependent services, and ever-changing user behavior.

One of the most important signs of an engineer's maturity is the shift from blind confidence in "personal ability" to deep trust in "robust processes." They gradually understand that what keeps a complex system running stably is not a genius engineer's "eagle eye," but a well-designed "immune system" that can automatically detect and fend off errors.

The goal of this "immune system" is to shift the basis for quality assurance from the subjective, error-prone "I think" to the objective, repeatable, data-driven "the system proves."

"I think it's fine" means the quality judgment depends on someone's experience and current state -- it is unreliable and untraceable.

"The system proves it's fine" means we have a series of automated tools, standardized processes, and quantitative data indicators that collectively "endorse" the quality of a release. This proof process is reliable, repeatable, and does not depend on anyone's state.

In Chapter 6, we already discussed the importance of "shifting testing left" -- using unit tests, continuous integration, and other means to find and fix issues as early as possible in the development phase. However, no matter how perfectly we execute in the development and testing phases, we cannot fully simulate the complexity and uncertainty of the production environment.

The real test always happens at the moment code is deployed to production, and every second thereafter.

Therefore, a complete quality assurance system must include a set of more rigorous and cautious "defense works" specifically designed for "production environments" and "production-like environments." This defense works consists of three key parts: the final line of defense before deployment -- the canary environment; the continuous guardian after deployment -- data patrols; and the last resort when unexpected events occur -- the emergency fix process.

In this chapter, we will delve into these three "production-grade" quality assurance practices and explore how to build a trustworthy delivery system that can self-prove and self-heal in a world full of risks.

Canary Anomaly Handling Standard Process: The Sentinel of Production-Like Environments

We mentioned earlier that "canary release" is one of the three pillars of high-quality releases. The canary environment, also known as the "canary environment," gets its name from an old tradition: miners would bring a canary down into the mine shaft. If the canary stayed healthy, it proved the air underground was safe.

In software releases, the canary environment is our "canary" sent into the "mine shaft" of the production environment. It is an independent online environment that is nearly identical to the formal production environment in terms of network, configuration, and infrastructure. Before a full release, we first deploy the new version of the code to this environment and direct a small portion of real, filtered production traffic to it.

The core value of canary release is that it gives us a valuable opportunity to verify the stability of a new version in the "real world" at the cost of a "minimum explosion radius."

However, having a canary environment alone is far from enough. If the team lacks a standardized, universally followed "canary verification and anomaly handling process," canary release can easily become a formality, or worse, create the mistaken mindset that "since we have canary, it doesn't matter if the code quality is a bit lower."

An effective canary process must be like a rigorous "scientific experiment." It has a clear "experimental goal," clear "observation metrics," and a "termination protocol" that must be immediately executed when "anomalous readings" appear.

Standard Operating Procedure (SOP) for Canary Release

Our team defines the entire canary release process as a six-step, checklist-based SOP. Each release must be checked and confirmed item by item by the designated release manager (usually the core developer of this release).

Step 1: Pre-Release Check

Before clicking "Deploy to Canary," the release manager must confirm the following:

• All related Merge Requests have been merged into the develop branch.
• The CI/CD pipeline is "green" on the develop branch (all automated tests and checks have passed).
• All changes included in this release have been clearly recorded in the Changelog draft.
• Have reached consensus with product and QA on the "verification focus" of this canary. That is, which new features or important fixes we need to pay particular attention to.
• Have posted a notification in the team's public release channel: "@all Starting canary release for v1.2.0. Core changes include [User Phone Login] and [Fix Payment Order Drop]. Release manager: @Zhang San."

Step 2: Deploy to Canary Environment

• Click the one-click deploy button through the release system.
• Monitor the deployment process, confirm all services have started successfully with no startup errors.

Step 3: Core Functionality Smoke Test

After deployment, the release manager and designated QA must immediately perform a quick "smoke test" of the system's "mainline" functionality in the canary environment. The purpose of this test is not to verify new features, but to ensure this change has not broken the system's most core, basic functions.

• Can users register and log in normally?
• Can core business flows (like placing an order, making a payment) be completed normally?
• Can the system's main pages be loaded normally without large-scale white screens or errors?

This smoke test should have a predefined, brief checklist.

Step 4: Observation and Data Comparison

After the smoke test passes, the canary phase enters its most core "observation period." This period typically lasts for a certain duration (e.g., 30 minutes to a few hours, depending on the complexity and risk of the change). During this time, the release manager must, like a commander sitting in a rocket launch control center, keep a close eye on the monitoring system's "dashboard."

The focus of observation is "comparison." We compare the metrics of the canary environment with those of the formal production environment during the same time period to see if they are "performing consistently."

• System metrics comparison: Are the CPU usage, memory usage, and network IO of the canary environment showing abnormal spikes or fluctuations?
• Application metrics comparison:
  • Error rate: Is the error rate (e.g., HTTP 5xx errors) in the canary environment significantly higher than in production? This is the most critical and dangerous signal.
  • Latency: Are the core interface response times (P99, P95) in the canary environment obviously increased compared to production?
• Business metrics comparison:
  • Conversion rate: For e-commerce, is the "order conversion rate" and "payment success rate" in the canary environment holding steady with production? Any small, unexplained drop could be a sign of trouble.
  • Engagement: For content products, are the "average time spent per user" and "content consumption" in the canary environment showing abnormal decreases?

Step 5: Anomaly Judgment and Decision

During the observation period, if any metric shows an "unexplained, significant negative deviation," the release manager must immediately initiate the "anomaly handling process."

The definition of "significant negative deviation" should be quantified. For example, the team can agree on:

• "The error rate in the canary environment is more than 0.1% higher than in production."
• "The P99 latency of core interfaces is more than 20% higher than in production."
• "The payment success rate is more than 0.5% lower than in production."

Once these thresholds are triggered, the release manager must make a binary decision: Abort or Fix Forward?

Abort: This is the default option. When an anomaly occurs, the safest and most responsible action is to immediately roll back the canary environment code to the previous stable version. Then, calmly and offline, investigate the cause.

Fix Forward: This is only allowed in very rare cases. The condition is: the cause has been 100% identified, and the fix is extremely simple, clear, and very low risk (e.g., modifying a configuration item). In this case, the fix can be quickly deployed to the canary environment and observation restarted.

This decision process must be transparent. The release manager should synchronize the observed anomaly, their judgment, and their final decision in the release channel in real-time. For example: "@all [Canary Anomaly] Observed a 1% drop in the payment success rate in the canary environment. [Decision] Roll back canary version immediately. I will investigate the cause offline."

Step 6: Canary Pass and Full Release

If all metrics are normal during the entire observation period and no anomaly thresholds are triggered, the canary release is judged to have "passed."

• The release manager announces in the release channel: "Canary passed. Ready for full release."
• Only after receiving a "confirmation" from at least one other team member (e.g., QA lead or tech lead) can the full release be executed. This "two-person confirmation" mechanism is the last safeguard against personal misjudgment.

This standardized process may seem tedious, but what it brings is enormous certainty and security. It transforms a process full of uncertainty and reliant on individual heroism -- an "art" -- into a repeatable, data-driven, collectively responsible "science."

It is like a loyal "sentry," conducting the final, most rigorous interrogation of our code before it steps into the vast production environment. Only changes that can provide an irrefutable "pass" of "the system proves it's fine" are allowed through.

Data-Driven Inspection: Using Scripts to Guard Data Consistency

Canary release primarily guards the system's "availability" and "performance." However, as emphasized in Chapter 6, "data consistency, accuracy, and reliability" are an equally important quality dimension, yet much harder to detect through real-time monitoring.

Many data problems are "silent killers." They don't cause system errors or slow down interfaces. They just quietly erode the foundation of your data in unseen corners.

A payment callback logic bug might cause a small number of orders to have their status eternally stuck at "Pending Payment" even after the user has paid successfully.

A concurrency issue in user points calculation might lead to a mismatch between the user's point balance and their point transaction records.

A failed data migration might result in subtle but fatal discrepancies between user data in the old and new systems.

These problems are extremely difficult to detect through macroscopic business metric monitoring alone. By the time user complaints come in, or a month-end financial reconciliation reveals them, severe and often irrecoverable damage has already been done.

Therefore, we must establish a proactive, regular, automated "data patrol" mechanism. This mechanism is like the "auditor" we station in the data warehouse, whose sole duty is to tirelessly check our data against predefined "rules" to see if it is still in a "self-consistent" and "healthy" state.

This "auditor" is not a person, but a set of automated patrol scripts.

The Core Idea of Data Patrol Scripts

The core idea of data patrol is based on a simple belief: in a healthy data system, certain "identity relationships" must exist between different data entities, and between different states of the same data entity. The job of the patrol script is to periodically verify whether these "identity relationships" still hold.

Common "identity relationships" and patrol script examples:

Total Amount Reconciliation

• Identity relationship: "Today's total amount of successful payments from all channels" should equal "today's total amount of all successfully paid orders."
• Patrol script: Every day at midnight, the script automatically calculates yesterday's total amount from both the payment gateway logs and the order database. Then it compares the two numbers. If they are not equal, even by a single cent, it immediately triggers the highest level of alert.
• Value: This script is extremely effective at detecting "order drop" or "double counting" issues in the payment process.

Status Consistency Check

• Identity relationship: "An order with a status of 'Completed' must have an associated payment record with a status of 'Successful'."
• Patrol script: Runs every hour, scans all orders updated to "Completed" in the past hour, and checks if each has a corresponding successful payment record. If any are missing, it records these "orphan order" IDs in the alert log.
• Value: This script can detect status inconsistency issues caused by lost async messages, failed callback handling, etc.

Transaction and Balance Reconciliation

• Identity relationship: "Any user's current point balance" should equal "their initial points + sum of all point addition transactions - sum of all point deduction transactions."
• Patrol script: Randomly samples 1% of users every day (this ratio is only an example), recalculates their current balance based on their point transaction records, and compares it with the balance field stored in the database. If there's a discrepancy, it triggers an alert.
• Value: This script is the "mirror" for uncovering issues like concurrent computation errors or precision loss that lead to imbalanced account balances.

Cross-System Data Sync Check

• Identity relationship: "The 'VIP user' list in the CRM system" should be completely consistent with "the list of users who have spent over 10,000 RMB cumulatively in the order system."
• Patrol script: Runs once a day, pulls user lists from both systems, and performs a set "difference" operation. If the difference is not empty, it indicates data sync delay or error.
• Value: This script can ensure eventual consistency of data between different systems in a complex, multi-microservice architecture.

Building an Effective Data Patrol System

Treat Patrol Scripts as "First-Class Citizens"

Patrol scripts are not disposable, one-time tools. They are "sentinels" guarding our core assets. They should be treated like our business code: version-controlled, code-reviewed, and equipped with complete documentation and comments.

Establish a Dedicated "Patrol Scheduling Platform"

As the number of patrol scripts grows, we need a unified platform to schedule, execute, monitor, and alert on them. This can be based on simple Cron jobs or more professional scheduling tools (like Airflow, Azkaban).

Alerts Must Be "Actionable"

A good alert doesn't just tell us "there's a problem," but provides enough information to help us quickly locate and resolve it. Patrol script alerts should include:

Which patrol rule was triggered?

What are the inconsistent data IDs or samples?

A link to the relevant documentation or handling plan.

Integrate "Writing Patrol Scripts" into the Development Process

The ideal state is: when developing a new feature that involves changes to core data, we should simultaneously think about and write the corresponding "data patrol scripts."

For example, when developing a new "coupon" feature, you should think about writing a patrol script to verify that "the number of used coupons" equals "the number of orders that used this coupon."

This is an extension of the "defensive programming" philosophy to the data layer.

Data patrol is a kind of "dumb" effort and "hard" work. Unlike developing a cool new feature, it doesn't bring an instant sense of accomplishment. Its value lies in those peaceful days when "nothing happens."

It uses a systematized, trusting-no-one approach to answer the most fundamental question: "Is our data OK?"

In a data-driven culture, the answer to this question cannot come from anyone's "feelings," but only from the cold, continuously running, never-slacking proof of "scripts."

Production Emergency Fix Process: Staying Graceful in the Storm

No matter how thorough our preventive measures are, the "storms" of the production environment -- those unforeseen, severe online incidents -- will inevitably arrive at some point.

In the moment of a storm, a team's collaboration level, professionalism, and cultural depth face their most severe test.

A chaotic team will descend into panic and blame. People will act like headless flies, each fighting alone, information opaque, decisions arbitrary, causing the problem to escalate.

A mature team will act like a well-trained "fire department." They have clear roles, standardized processes, and calm minds. Under immense pressure, they can methodically contain the fire, eliminate the hazard, and ultimately minimize the damage.

This ability to "stay graceful in the storm" is not innate. It comes from repeated drills beforehand and a set of "emergency fix processes" (Hotfix Process) imprinted on everyone's bones.

The goal of this process is to find the best balance between the seemingly contradictory goals of "speed" and "safety." We need to restore user service as quickly as possible while ensuring that our fix itself does not introduce new, more serious problems.

Emergency Fix Process (Hotfix SOP)

When a severe online incident (which we call "P0" or "P1" level) is confirmed, the team immediately suspends all non-urgent routine work and initiates the following SOP:

Phase 1: Emergency Response and Information Sync

Establish Emergency Response Team

Immediately create a temporary IM group or video conference room dedicated to this incident.

Designate a clear "Incident Commander" (IC). The IC's role is not to personally fix the problem, but to coordinate resources, collect information, make decisions, and sync progress externally. This role is typically filled by a senior tech lead or architect.

Pull in all relevant technical staff (development, operations, QA) and non-technical staff (product, customer support, operations) into this "war room."

Establish a "Single Source of Truth"

The IC is responsible for publishing a "situational report" to everyone in the "war room" at fixed intervals (e.g., every 15 minutes).

The report should include:

• Current status: Is the problem still ongoing? Is the scope expanding or shrinking?
• Investigation progress: What investigative methods have we tried? What new findings are there?
• Next steps: What do we plan to do next? Whose help is needed?

This mechanism greatly reduces information noise, preventing technical staff from being repeatedly interrupted by unrelated people asking "What's happening?"

Prioritize "Stopping the Bleeding" Over "Curing the Disease"

In the early stages of incident handling, our primary goal is to restore the core functionality of the service as quickly as possible, even if it is only a temporary, imperfect solution.

Common "stop the bleeding" methods include:

• Rollback: If the incident was caused by a recent release, the fastest and safest way to stop the bleeding is to immediately roll back to the previous stable version.
• Degrade: Turn off or degrade some non-core functions of the system (e.g., turn off the recommendation system and only keep the search function) to ensure the stability of core functions.
• Rate Limit: If the incident is caused by a sudden traffic surge, temporarily enable rate limiting, sacrificing some users to protect the entire system.

Phase 2: Root Cause Location and Code Fix

Conduct Root Cause Analysis After "Stopping the Bleeding"

Only after the core functions of the service have been restored can we relatively calmly dig into the root cause of the problem.

Create a "Hotfix" Branch

Once the root cause is identified and a code fix is needed, the developer must create the emergency fix branch from the master branch (the branch representing production code), not from develop.

Branch naming convention: hotfix/[ticket-id]-[short-description]. For example: hotfix/TICKET-789-fix-null-pointer-on-user-profile.

Why must it be created from the master branch? Because the develop branch may contain other unreleased, unstable new features. We don't want to introduce these unrelated, risky changes in an emergency fix.

The Fix Must Be "Minimal"

Hotfix changes must follow the "principle of minimality." Only modify the minimum amount of code directly related to this incident.

It is strictly forbidden to include any "optimizations" or "refactoring" unrelated to this incident in a Hotfix. This behavior, known as "hitching a ride," is a major taboo in emergency fixes.

Accelerated but Not Skipped Review

Hotfix code must also undergo code review.

However, the review process can be accelerated. For example, it can be expedited by requiring at least one senior engineer to conduct an "urgent" review.

The focus of the review is to confirm the correctness of the fix and whether it might introduce new side effects.

Phase 3: Emergency Release and Verification

Hotfix Release Must Also Follow the "Canary" Process

Even in an emergency, we cannot directly release the Hotfix to all production users at once. We must still first deploy it to the canary environment for small-traffic verification.

The canary observation time can be shortened (e.g., shortened to 5-10 minutes).

But the core metric observation and smoke test cannot be skipped.

Merge and Sync

After canary verification passes, release the Hotfix to the full production environment.

After the release is successful and the problem is confirmed to be resolved, two things must be done immediately:

a. Merge the hotfix branch back into the master branch and tag it with a new version (e.g., v1.2.1). b. At the same time, also merge the hotfix branch back into the develop branch. This step is crucial but often forgotten. If not done, this fixed bug will be brought back online in the next normal release.

Phase 4: Postmortem and Improvement

The end of incident handling is not when the service is restored, but when we have learned something from this incident and truly made improvements.

Within 1-2 days after the incident is resolved, the IC must organize a "blameless postmortem meeting."

The ultimate output of the postmortem must be a series of actionable "improvement items" with clear owners and deadlines, aimed at preventing similar problems from occurring again at the system and process level.

This standardized emergency fix process is like a set of "emergency plans." It cannot prevent the storm from coming, but it ensures that when the storm hits, everyone on our ship knows their position, understands their duties, and can work together to get through the storm, rather than being consumed by chaos.

Summary: Quality Is Designed, Not Tested

In this chapter, we explored three "production-grade" quality assurance practices: canary process, data patrol, and emergency fix. Together, they form a defense-in-depth system aimed at guarding the ultimate value we deliver to users.

These processes may seem complex, even somewhat "bureaucratic." But behind them runs a unified, profound philosophy:

High quality is designed, not tested.

It is designed when we plan the canary process, pre-designing our "response patterns" for facing uncertainty.

It is designed when we write data patrol scripts, pre-designing the "rule system" for verifying data consistency.

It is designed when we create the emergency fix SOP, pre-designing the "organizational capability" for responding to crises.

These "designs" spread the responsibility for quality assurance from the shoulders of a few "QA" individuals to every stage of the entire R&D process and into the heart of every role.

What it ultimately aims to achieve is a cultural shift: from "we hope the system is fine" to "we have the ability to systematically prove to ourselves and to users that our system is fine."

This leap from "hope" to "proof" is the "coming of age ritual" that marks the transition from an amateur team to a professional one.