祝融说。

24.1 Three Lines of Quality Defense: Prevention, Inspection, Audit

Terminology distinction: The "three lines of quality defense" here refers to the three layers of quality governance (prevention/inspection/audit), which is a different concept from the "three lines of defense" in the acceptance stage of Section 9.2 (functional/architectural/security acceptance). Note the distinction.

A common scenario (a composite case illustrating the failure pattern): a team adopts AI coding, productivity multiplies, but a month later the codebase is a mess -- not because AI is bad, but because there were no quality gates.

You let the team use AI coding, productivity rose, and you were satisfied. But a month later, you find the codebase has become a mess -- inconsistent style, severe module coupling, and some core functionality quietly changed by AI. You ask the team: "What happened?" They answer: "AI wrote it, we didn't look closely."

This is the consequence of having no quality gates. The "side effects" of AI coding -- surging code volume, uneven quality, architectural drift -- are not problems with AI itself, but "a lack of process". Quality gates exist to solve this: set up checkpoints along the path where code flows toward production, ensuring every segment of code is verified.

Quality governance is not "one stage" but "three layers" -- prevention, inspection, audit -- each layer covering blind spots the previous layer cannot reach.

With only prevention, you cannot discover problems during coding; with only inspection, you cannot discover systemic trends. The three defense lines complement each other; none can be omitted.

First defense line: Prevention.

Prevention has the lowest cost and the best effect; it ensures the direction is correct before coding begins:

• Blueprint review: Before each project begins, the blueprint must be reviewed by at least one colleague. The review focuses on: whether terminology is accurate, whether the data model is complete, and whether milestone division is reasonable. Coding can begin only after the review passes.
• Acceptance criteria up front: Before each milestone begins, acceptance criteria are already defined, written into the milestone description rather than thought up ad hoc after coding is done. Acceptance criteria should be verifiable (not "good code quality" but "code passes the Lint check").

Second defense line: Inspection.

Inspection is the quality gate after coding; it is the most critical stage in AI coding:

• Mandatory acceptance: All AI-generated code must pass acceptance before it can be submitted; acceptance results must be recorded (pass/fixed/rebuild); acceptance records serve as a reference for code review.
• Three acceptance principles: functional acceptance (does the code implement the requirements?), architectural acceptance (does the code deviate from the blueprint?), and security acceptance (does the code have security risks?).

Third defense line: Audit.

Audit is a periodic retrospective check that uncovers systemic problems:

• Audit frequency: small projects are audited once at project end; large projects are audited once a month.
• Audit content: blueprint coverage (how many projects have blueprints?), acceptance execution rate (how many milestones have acceptance records?), architectural drift rate (how much code has architectural drift?), technical debt assessment (how healthy is the codebase?).

24.2 Four Quality Gates: Local Acceptance → Code Review → Integration Verification → Deployment

Quality gates are automated or semi-automated quality checkpoints set along the path where code flows toward production. Why four gates? Because each gate solves a different problem: the local acceptance gate solves "was it done seriously", the code review gate solves "was it done correctly", the integration verification gate solves "does it work together", and the deployment gate solves "can it go live".

Gate 1: Local acceptance gate.

• Location: developer's local environment, after AI completes coding;
• Check items: functional completeness check, architectural compliance check, security check (basic);
• Pass condition: all checks pass, or NEEDS_FIX has been resolved.

Gate 2: Code review gate.

• Location: before submitting to a shared branch;
• Check items: code style check (automated), test coverage check (automated), code review (manual);
• Pass condition: automated checks pass + at least one colleague's review passes.

Gate 3: Integration verification gate.

• Location: before merging into the main branch;
• Check items: build check (can the project build successfully), test suite (all tests pass), integration test (cross-functional verification);
• Pass condition: all checks pass.

Gate 4: Deployment gate.

• Location: before deploying to production;
• Check items: security review (comprehensive), performance testing (if needed), change record check;
• Pass condition: all checks pass.

24.3 Standard Operating Procedure (SOP) for Canary Anomaly Handling

The gray-release environment, also called the "canary environment" -- the name comes from an old tradition: before miners descended into a shaft, they would first lower a canary into the mine; if the canary remained unharmed, it proved the air below was safe. In software releases, the canary environment is the "canary" we send into the "mine" of production. It is an independent online environment that is nearly identical to the official production environment in network, configuration, and infrastructure; before a full release, we first deploy the new version's code into it and route a small portion of genuine, filtered production traffic into it.

The core value of canary release: verifying the new version's stability in the "actual world" at the cost of the "minimum blast radius". But a canary environment alone is far from enough -- if the team lacks a standardized "canary verification and anomaly-handling process", the canary may become a mere formality, and even breed the wrong mindset of "with a canary in place, slightly worse code quality doesn't matter".

An effective canary process must resemble a rigorous "scientific experiment": with a clear "experimental objective", well-defined "observation metrics", and an "abort protocol" executed immediately when an "abnormal reading" appears. The following six-step SOP is a standard process honed through practice:

Step 1: Pre-release checks       Confirm merge, green CI, changelog, verification
                                priorities, and public-channel notification.
Step 2: Deploy to canary         One-click deployment; confirm service startup.
Step 3: Core regression checks   Smoke-test registration/login, core business
                                workflows, and main pages.
Step 4: Observe and compare      Compare canary vs. production: system metrics,
                                error rate, latency, and business metrics.
Step 5: Assess anomalies         Threshold triggered → abort (default) or fix
                                forward (very rarely).
Step 6: Pass and release fully   Announce the pass + two-person confirmation
                                → full release.

Step 4, "Observation and data comparison" is the core of the entire canary process: comparing whether the canary environment's metrics and the production environment's metrics "perform consistently" over the same time window -- system metrics comparison (do CPU, memory, and network IO spike abnormally), application metrics comparison (error rate is the most critical and dangerous signal; do P99/P95 latencies increase noticeably), and business metrics comparison (do conversion rate, payment success rate, and per-capita usage duration drop abnormally).

In Step 5, "Anomaly judgment and decision", the "significant negative deviation" must be quantified, for example: "the canary environment's error rate is more than 0.1% higher than production", "the core interface's P99 latency is more than 20% higher than production", "the payment success rate is more than 0.5% lower than production". Once a threshold is triggered, the release owner must make a black-and-white decision:

• Abort -- the default option: immediately roll back the canary code to the last stable version, then calmly investigate the cause offline;
• Fix Forward -- rare cases: allowed only when the problem's cause has been 100% located, the fix is extremely simple and clear, and the risk is extremely low (such as changing a single configuration item).

This decision process must be open and transparent -- the release owner should sync the observed anomalies, judgments, and final decision in real time in the release channel.

Step 6, "Canary pass and full release": if all metrics are normal during the observation period, it is judged as "passed". After the announcement is sent, you must obtain "confirmation" from at least one other team member before executing the full release -- the "two-person confirmation" mechanism is the last line of insurance against individual misjudgment.

24.4 Data Is a Red Line: Consistency, Accuracy, Reliability

Canary release mainly guards the system's "availability" and "performance". However, the "consistency, accuracy, and reliability" of data is another quality dimension that is equally important yet harder to detect through real-time monitoring.

Many data problems are "silent killers". They do not cause system errors, nor do they slow down interfaces; they just quietly erode the foundation of data in corners you cannot see:

• A logic bug in a payment callback may cause a small portion of orders to remain stuck in "pending payment" forever after the user has paid successfully;
• A concurrency issue in points calculation may cause a user's points balance to fail to match the points transaction log;
• A failed data migration may cause subtle yet fatal discrepancies in user data between the old and new systems.

These problems are extremely hard to detect through macro-level monitoring; by the time they surface through user complaints or month-end financial reconciliation, serious damage has often been done. Therefore, you must establish a proactive, periodic, automated "data patrol" mechanism -- an indefatigable "auditor" stationed inside the data warehouse: patrol scripts.

The core idea of data patrol: in a healthy data system, between different data entities, and between different states of the same entity, certain "invariant relationships" must hold. The job of patrol scripts is to periodically verify whether these "invariant relationships" still hold.

Common "invariant relationships" and patrol scripts:

Key points for building an effective data patrol system: treat patrol scripts as "first-class citizens" (put them under version control, code review, and thorough documentation just like business code); build a unified "patrol scheduling platform" (Cron Job or Airflow, etc.); alerts must be "actionable" (which rule was triggered, the inconsistent data IDs/samples, a link pointing to the handling playbook); internalize "writing patrol scripts" into the development process -- when developing new features that involve changes to core data, think about and write the accompanying patrol scripts at the same time.

Data patrol is "plodding" work, and "grueling" work too. Its value shows up in those quiet days when "nothing happened" -- it answers, in a systematic way that trusts no one, that most fundamental question: "Is our data okay?" The answer to this question cannot come from anyone's "feeling"; it can only come from the "proof" of cold, continuously running, untiring scripts.

24.5 Fail Early: Shift-Left Testing, Exploratory Testing, and a Culture That Embraces "Failure"

No matter how thorough the preventive measures, accidents will happen. Faced with this uncertainty, the core belief of mature teams is: fail early -- the earlier a problem is exposed, the smaller the cost.

The "fail early" strategy means installing "safety valves" at every stage of the development process, so that problems surface as close to their source as possible and at minimum cost:

• Shift-left testing: move testing from "a final check after development is done" to earlier in the development process. Unit tests are written alongside coding, and integration tests are run before features are merged -- the earlier a problem is found, the lower the fix cost (this is an extension of the same economic law that "changing requirements during the coding phase costs 10 times as much as during the analysis phase").
• Exploratory testing: do not only run pre-defined test cases. Encourage testers and engineers to "break" the product like genuine users -- entering bizarre data, clicking rapidly in succession, and walking paths no one has taken. Many of the deepest-hidden problems are discovered through exploration, not through planning.
• A culture that embraces "failure": failing early means the team must treat "discovering a bug" as "saving a future fix cost" rather than "a failure". Only when failure is safe and learnable will team members proactively look for potential problems instead of covering them up.

Continuous integration (CI) is the automation engine of "fail early": every code commit automatically triggers a build and tests; any line of code that breaks existing functionality is exposed within minutes by the red-green CI status, rather than being discovered days after merging, or even weeks after going live.

For the "fail early" philosophy to truly take root, it ultimately needs cultural support: when the team has built a psychological safety zone where "failure is welcomed, because it lets us solve problems earlier and more cheaply", quality changes from a "defensive move" into an "offensive weapon".

Supplement: when a production-environment storm actually arrives, a standardized "Hotfix SOP" is also needed -- form an incident response team, designate an Incident Commander (IC, responsible for coordination rather than fixing things personally), establish a single source of truth with a "situation briefing" every 15 minutes, prioritize "stopping the bleeding" over "curing the root cause" (rollback/degradation/rate limiting), create a hotfix branch from the master branch, the fix must be "minimalized" and strictly forbidden from "hitchhiking" unrelated optimizations, and accelerate but never skip the review. The complete Hotfix SOP is in Appendix E.