祝融说。

In Chapter 9, we invested significant effort in learning how to design and install a precision "nervous system" -- structured logging and full-chain tracing -- for our software system. We have successfully transformed those fleeting, invisible "events" in the real environment into tangible, persistent, structured "data."

Now, we hold this valuable "medical report" in our hands and have entered the most critical step in the entire "telemetry-driven development" process: how to effectively present this report to our "AI attending physician" and guide it to make an accurate diagnosis.

I call this process "reverse feeding." Traditional AI programming involves us (humans) feeding "intent" (requirements) to AI, and AI produces "code." "Reverse feeding," on the other hand, involves feeding the "reality" (log data) captured from the real environment back to AI, and AI produces "insight" (root cause analysis).

In this chapter, we will completely change the way you report bugs to AI. You will learn how to transform from a "patient's family member" who can only vaguely "orally describe" symptoms, into a "professional forensic examiner" who can provide precise, quantifiable, unambiguous "pathological slides." Through a complete real-world scenario, we will walk through the entire "black-box troubleshooting" process of human-machine collaboration, from problem discovery to the final fix.

10.1 Stop Describing Verbally -- Throw the Crash Log at It Directly

Imagine you go to see a doctor. Which of the following two descriptions would help the doctor solve your problem faster?

• Description A (Verbal): "Doctor, I've been feeling a bit off lately. I get dizzy sometimes, and my stomach doesn't feel great either."
• Description B (Data-Driven): "Doctor, here are my blood pressure, heart rate, and temperature readings from the past week. I also have my blood test results. My symptoms usually occur about two hours after lunch."

The answer is obvious. Description A is full of subjective, vague, unquantifiable information. The doctor needs to spend significant time playing "twenty questions" to eliminate possibilities. Description B provides the doctor with objective, quantified data that includes time and context. The doctor can immediately look for "abnormal patterns" in it, greatly shortening the diagnostic path.

The same principle applies when we report bugs to AI.

The Inefficient "Oral Description" Mode

This is the approach most developers (including my past self) instinctively use, and it is also the least efficient.

You (to AI): "My users report that when they upload a large file, sometimes the progress bar gets stuck at 99%, and then it fails, but sometimes it works. The code looks fine to me. Can you help me analyze what the possible causes might be?"

This paragraph is a classic example of "oral description." It commits several fatal mistakes:

1. Full of "uncertain" words: "sometimes," "might," "seems like." These are huge sources of noise for a probabilistic AI model.
2. Contains "subjective judgment": "The code looks fine." This is a highly misleading, unverified assumption that might directly trigger AI's "confirmation bias," shifting its attention away from the code itself.
3. Lacks "reproducible" context: "Upload a large file" -- how large? What format? What was the user's network environment? Which server was it uploaded to? All these critical environmental variables are missing.

Faced with such a description, what AI can do is extremely limited. It can only act like a search engine, listing a bunch of generic "common causes of file upload failure": network timeout, server configuration limits, insufficient disk space, frontend script errors... This list might be useful, but it is still miles away from your specific bug. You still need to manually investigate each possibility yourself.

The Efficient "Reverse Feeding" Mode

Now, let us switch to "telemetry-driven" mode. Because we have already preemptively implanted good structured logging in the system, when a problem occurs, the first thing we do is not "guess," but "extract."

We extract from the logging system the complete log stream associated with that failed upload operation, identified by the same trace_id. Then, we "throw" this original, unaltered data directly at AI.

You (to AI):

Context: We are investigating a file upload failure. I have extracted the complete, structured log stream for a failed operation, identified by trace_id: "trace-xyz-789".

Your Role: Act as a Senior Site Reliability Engineer (SRE). Your task is to analyze these logs, identify the root cause of the failure, and propose a specific solution.

Log Data:

[
{"timestamp": "10:30:01Z", "level": "INFO", "event": "FILE_UPLOAD_STARTED", "trace_id": "trace-xyz-789", "context": {"user_id": "user-123", "filename": "report.pdf", "filesize": 52428800}},
{"timestamp": "10:30:02Z", "level": "INFO", "event": "AUTH_CHECK_SUCCEEDED", "trace_id": "trace-xyz-789", "context": {"service": "upload-gateway"}},
{"timestamp": "10:30:03Z", "level": "DEBUG", "event": "CHUNK_UPLOAD_STARTED", "trace_id": "trace-xyz-789", "context": {"chunk_index": 0, "chunk_size": 1048576}},
// ... a lot of successful chunk uploads ...
{"timestamp": "10:31:28Z", "level": "DEBUG", "event": "CHUNK_UPLOAD_SUCCEEDED", "trace_id": "trace-xyz-789", "context": {"chunk_index": 48}},
{"timestamp": "10:31:30Z", "level": "DEBUG", "event": "CHUNK_UPLOAD_STARTED", "trace_id": "trace-xyz-789", "context": {"chunk_index": 49}},
{"timestamp": "10:31:35Z", "level": "WARN", "event": "S3_PUT_OBJECT_TIMEOUT", "trace_id": "trace-xyz-789", "context": {"bucket": "user-uploads", "key": "report.pdf.part49", "retry_attempt": 1}},
{"timestamp": "10:31:45Z", "level": "WARN", "event": "S3_PUT_OBJECT_TIMEOUT", "trace_id": "trace-xyz-789", "context": {"bucket": "user-uploads", "key": "report.pdf.part49", "retry_attempt": 2}},
{"timestamp": "10:32:00Z", "level": "ERROR", "event": "CHUNK_UPLOAD_FAILED", "trace_id": "trace-xyz-789", "context": {"chunk_index": 49, "reason": "Max retries exceeded"}},
{"timestamp": "10:32:01Z", "level": "INFO", "event": "UPLOAD_ABORTED", "trace_id": "trace-xyz-789"},
{"timestamp": "10:32:02Z", "level": "ERROR", "event": "FILE_UPLOAD_FAILED", "trace_id": "trace-xyz-789", "context": {"reason": "Chunk upload failed at index 49"}}
]

Analysis Request: Based solely on the provided log data, please answer:

1. What is the precise point of failure?
2. What is the most likely root cause?
3. Propose a code-level fix for the identified service.

Why is this approach so efficient?

1. Fact-driven, not opinion-driven: You did not give AI any subjective, potentially biased "guidance." You gave it cold, objective "evidence." This forces AI's analysis to be based entirely on data, not its "general knowledge."
2. Self-contained context: This log already contains all the key information AI needs to make a diagnosis: the start of the operation (FILE_UPLOAD_STARTED), file size (filesize), chunked upload logic (CHUNK_UPLOAD), interaction with external dependencies (S3_PUT_OBJECT), retry logic (retry_attempt), and the direct cause of the final failure (Max retries exceeded).
3. The problem is precisely "located": AI does not need to guess whether the problem is on the frontend, backend, or network. The log clearly shows that the problem occurred in the backend service, during the upload of the 50th file chunk to S3 storage, when consecutive timeouts occurred.

Based on this data, AI's diagnosis will be quite precise (its hypothesis still needs your verification in Step 4):

AI (SRE Mode) Response:

1. Point of Failure: The failure occurred precisely during the upload of the 50th chunk (index 49) of the file report.pdf. The direct cause was the CHUNK_UPLOAD_FAILED event, triggered after two consecutive S3_PUT_OBJECT_TIMEOUT warnings.

2. Most Likely Root Cause: The logs strongly indicate an issue with the interaction between our backend service and the S3 storage service. Given that the first 49 chunks (approx. 49MB) uploaded successfully over more than a minute, and then failures started occurring with timeouts, the root cause is unlikely to be a persistent network issue. The most probable causes are:

• S3-side throttling: The S3 bucket might have a rate limit policy that we are hitting after a certain number of PutObject operations in a short time.
• Client-side timeout configuration: The timeout setting in our S3 client SDK might be too aggressive. For large files, network conditions can fluctuate, and a longer timeout might be required for the final chunks. The default SDK timeout is often 60 seconds, and our logs show retries happening at 10 and 15-second intervals, suggesting a much shorter custom timeout.

3. Proposed Fix: I recommend increasing the S3 client's connection and socket timeouts within the upload service.

# Example in Python using Boto3
from botocore.config import Config

# Increase timeout from default 60s to 300s
s3_config = Config(
  connect_timeout=300,
  read_timeout=300,
  retries={'max_attempts': 5}
)
s3_client = boto3.client('s3', config=s3_config)

Additionally, we should implement an exponential backoff strategy for retries, instead of the current fixed-interval retries, to better handle transient network issues or S3 throttling.

Compare the output of the two modes. In the former, you get a "list of possibilities," and the work has just begun. In the latter, you get an actionable, code-level fix with a reasonable probability of being correct, and the work is more than half done -- the closed-loop verification of section 10.3 still follows.

This is the power of "reverse feeding." Stop "chatting" with AI about bugs; start "feeding" it data.

10.2 Identifying Illusions in Logs: Unnecessary Fallbacks That Mask the Real Problem

Directly feeding logs, while powerful, is not foolproof. A poorly designed system's logs themselves can also "lie." The most common "lies" come from overly broad, undifferentiated "error handling" and "degradation logic" in the code.

A robust system should try to keep running when errors occur. This is usually achieved through try...catch blocks and degradation mechanisms. For example, a recommendation system that cannot connect to its personalization engine might "degrade" to showing a generic "popular items" list.

This mechanism is good for "user experience," but it can be a disaster for "problem diagnosis." Because it masks the real root cause of the error, covering up a serious problem that should have been an ERROR with a seemingly "normal" INFO log ("Fallback to generic recommendations").

Typical Patterns of Log Illusions

• Pattern One: The "Catch-All" Block

try {
  // ... complex business logic that may contain multiple different failure causes
} catch (error) {
  log.error({ event: "OPERATION_FAILED", error: error.message }); // Lost error type and stack
  return genericFallbackResponse();
}

This catch block flattens completely different types of errors -- "database timeout," "third-party API authentication failure," "null pointer exception" -- into a single vague OPERATION_FAILED log. When AI sees this log, it loses all the key information needed to determine the root cause.

• Pattern Two: Silent Failure

def update_user_cache(user_id, data):
  try:
    redis_client.set(f"user:{user_id}", data)
  except RedisError as e:
    # The cache update failed, but the app can continue.
    # So, we just log a warning and move on.
    log.warn({"event": "CACHE_UPDATE_FAILED", "user_id": user_id})
    # No error is re-thrown. The caller doesn't even know it failed.

This "silent failure" leads to a very strange phenomenon: the main flow logs look completely normal, and the user operation "succeeds." But in reality, an important part of the system (the cache) is already in an inconsistent state. When subsequent operations depend on this cache that should have been updated, inexplicable, hard-to-reproduce bugs appear. AI analyzing the main flow logs will be completely misled.

How to Train AI to Be a "Log Detective"

In the face of potentially "lying" logs, we need to guide AI to upgrade from a mere "log reader" to a "log detective" capable of identifying "subtext" and "hidden information."

Technique One: Ask AI to Look for "Pattern Breaks"

After feeding the logs, add this instruction:

"In addition to finding errors, analyze the sequence of events. Are there any expected INFO logs that are missing? Is there a point where the log pattern naturally breaks from the typical success-case pattern?"

This instruction guides AI to look for things that "should have happened but did not." For example, in a successful flow, a CACHE_UPDATE_STARTED log is always followed by a CACHE_UPDATE_SUCCEEDED log. If, in a failed trace, the STARTED log appears but the SUCCEEDED log is missing, replaced by an seemingly unrelated WARN log, AI can keenly capture this "pattern break" and infer that the cache update was likely "silently" failed.

Technique Two: Cross-Validate Code and Logs

When you suspect the logs are "illusory," feed the relevant code snippet together with the logs to AI.

Context: I'm seeing a generic OPERATION_FAILED log, but I suspect it's hiding the real issue.

Log Snippet: [... the generic error log ...]

Relevant Code:

// ... the "catch-all" try-catch block ...

Analysis Request: Based on the provided code, list all the potential, specific errors that could be caught by this try...catch block and then logged as the generic OPERATION_FAILED message. Which of these potential errors is the most likely, given our current problem description?

This instruction forces AI to switch from "log analysis" to "static code analysis." It will read the code in the try block, identify all the places where exceptions could be thrown (database calls, API requests, JSON parsing, etc.), and then generate a list of "real failure causes." This is like giving the detective a list of suspects to compare against the on-site evidence.

Technique Three: Actively Ask About "Degradation Paths"

"Does this log stream suggest that any system fallback or graceful degradation logic was triggered? If so, what was the original failure that triggered this fallback?"

This question directly focuses AI's attention on the core task of "identifying illusions." It prompts AI to look for WARN level logs, or logs whose message contains keywords like "fallback," "generic," or "default," and correlate them with prior ERROR events.

A qualified AI collaborator cannot blindly trust the logs. You must always maintain a healthy "skepticism" and learn to use the above techniques to guide AI in penetrating the surface of the logs and excavating the original "first crime scene" obscured by clumsy error handling logic.

10.3 Forming a Closed Loop: Run on Real Hardware, Collect Logs, Feed AI, Root Cause Inference, Verify

We have already mastered the key skills of "evidence collection" (telemetry) and "analysis" (feeding). Now, it is time to string them together into a complete, repeatable, efficient "human-machine collaboration troubleshooting closed loop."

This closed loop will completely change your mindset when facing complex bugs. You will no longer feel helpless and anxious. Instead, you will systematically execute a standardized diagnostic process, like an experienced doctor.

The Five Steps of the Closed Loop:

Step 1: Reproduce and Trigger

• Goal: Stably reproduce the problem on a "real machine" (or staging environment) as close to production as possible.
• Your Role: "Test Engineer."
• Key Actions:
• If the problem is intermittent, try to find the specific conditions that trigger it (specific user, specific data, specific operation sequence).
• Before triggering the problem, ensure your telemetry system is ready and the log level is temporarily raised to DEBUG (if needed) to capture the most detailed information.
• Execute the operation that causes the problem.

Step 2: Extract and Isolate

• Goal: Precisely extract the complete, clean log stream related to the failed operation from the vast sea of logs.
• Your Role: "Data Analyst."
• Key Actions:
• Find the trace_id for this operation based on the operation's timestamp, user_id, and other information.
• Use this trace_id to export all relevant logs from _STARTED to _FAILED from the log aggregation system.
• Save the logs as a separate JSON or text file. Do not manually modify or trim them; keep them original.

Step 3: Feed and Guide

• Goal: Feed the extracted log data to AI in a structured way, and use precise questions to guide its analysis.
• Your Role: "AI Interaction Expert" (the core role of this book).
• Key Actions:
• Use the "reverse feeding" template from Section 10.1.
• Explicitly assign AI an expert role (SRE, DBA, etc.).
• Paste the log data directly.
• Ask specific, closed-ended questions ("What is the root cause?", "What is the fix suggestion?").
• If you suspect the logs are illusory, use the "detective" techniques from Section 10.2 to follow up.

Step 4: Reason and Hypothesize

• Goal: Receive and evaluate the "root cause reasoning" and "fix hypothesis" given by AI.
• Your Role: "Senior Engineer / Architect."
• Key Actions:
• AI will give one or more "most likely" root causes. You need to use your domain knowledge and experience to judge which hypothesis is most logical and worth validating.
• AI is not a god; its hypotheses can also be wrong. At this point, your critical thinking is crucial. If AI's hypothesis seems unreliable, point out its logical flaws and ask it to propose a new hypothesis based on the logs.
• The final decision of "which hypothesis to adopt" must be made by you.

Step 5: Verify and Fix

• Goal: Transform the "hypothesis" adopted in the previous step into a concrete "experiment" and verify and fix it in code.
• Your Role: "Developer."
• Key Actions:
• Verify: Based on AI's hypothesis, design a minimal experiment to verify it. For example, if AI hypothesizes "the database connection pool is exhausted," write a simple script that frantically creates database connections in the test environment to see if it reproduces the same error. Verify before fixing!
• Fix: Once the hypothesis is verified, adopt AI's suggested (or your improved) fix, and modify the code.
• Regression test: After applying the fix, go back to Step 1 and repeat the operation that triggers the problem in the same environment to ensure the bug is truly fixed and no new issues have been introduced.
• Update logs: If you find during this troubleshooting process that logging is insufficient somewhere, making analysis difficult, improve the logging at that point while fixing the bug. This is a valuable practice of continuous improvement of "system observability."

This closed-loop process perfectly combines the strengths of humans and AI:

• AI's strengths: Ability to process massive, structured data; ability to perform rapid pattern matching and correlation analysis; vast, cross-domain general technical knowledge.
• Human's strengths: Deep understanding of specific business domains; critical thinking and intuition; courage and sense of responsibility to make final decisions under uncertainty.

Through this process, troubleshooting a "black box" problem will no longer be a headache-inducing, luck-based "guessing game," but a methodical, step-by-step, data-driven "scientific investigation."

[Troubleshooting Case Study] A Complete "Black Box" Crash Fix Walkthrough

Let us apply all the theory to an end-to-end walkthrough case (the case is a teaching composite, but the workflow and data shapes are drawn from real troubleshooting practice).

Background: A backend engineer at an e-commerce platform (the character is fictional). The operations team reports that the production Order Service experiences a few minutes of CPU spikes around midnight every day, accompanied by large numbers of API timeout errors. Then the service automatically restarts and returns to normal. No one knows why.

Step 1: Reproduce and Trigger (Run & Trigger)

This problem occurs regularly, so "reproducing" is relatively easy. We do not need to trigger it manually. Our task is to prepare for "evidence collection" before the next midnight arrives.

• Your Action: You log in to the production monitoring platform and confirm the Order Service log level is already INFO. You specifically check the modules related to "scheduled tasks" or "batch processing" and temporarily raise their log levels to DEBUG. You set up an alert to notify you when CPU usage exceeds 95%.

Step 2: Extract and Isolate (Extract & Isolate)

At 12:05 AM, your phone receives the alert. After experiencing a few minutes of "near-death," the service is automatically restarted by the container orchestration system (e.g., Kubernetes).

• Your Action: You immediately log in to the logging system, lock the time range to 23:55 to 00:05. You see thousands of log entries. You first filter for level: ERROR and find a lot of them read "Request timeout after 30s." You pick a timed-out request at random and get its trace_id. However, the logs in this trace only show the request coming in, and then nothing else, until the 30-second timeout. This path leads nowhere.
• You change your strategy. Instead of focusing on "failed requests," you look for what the system was "actively doing" during that time period. You search for logs whose event name contains JOB, TASK, or SCHEDULE.
• You find it! You spot an INFO log: {"timestamp": "00:00:01Z", "event": "NIGHTLY_REPORT_GENERATION_JOB_STARTED", ...}. Immediately following it are a large number of DEBUG logs, showing this Job frantically looping through some data. You isolate the complete log stream for this Job.

Step 3: Feed and Guide (Feed & Guide)

You copy all the relevant logs from JOB_STARTED until the point of service crash, and prepare to feed them to AI.

You (to AI):

Context: Our Order Service crashes every midnight due to 100% CPU usage. I've isolated the logs from a scheduled job that runs at that time.

Your Role: Act as a Senior Go Performance Engineer. Analyze these logs to find the performance bottleneck.

Log Data:

[
{"timestamp": "00:00:01Z", "level": "INFO", "event": "NIGHTLY_REPORT_JOB_STARTED", "job_id": "job-1"},
{"timestamp": "00:00:01Z", "level": "INFO", "event": "FETCHING_VIP_USERS", "job_id": "job-1"},
{"timestamp": "00:00:02Z", "level": "INFO", "event": "FETCHING_VIP_USERS_SUCCEEDED", "job_id": "job-1", "context": {"user_count": 50}},
{"timestamp": "00:00:02Z", "level": "DEBUG", "event": "PROCESSING_USER_START", "job_id": "job-1", "context": {"user_id": "vip-user-1"}},
{"timestamp": "00:00:02Z", "level": "INFO", "event": "FETCHING_USER_ORDERS", "job_id": "job-1", "context": {"user_id": "vip-user-1", "time_range": "last_365_days"}},
{"timestamp": "00:00:08Z", "level": "INFO", "event": "FETCHING_USER_ORDERS_SUCCEEDED", "job_id": "job-1", "context": {"user_id": "vip-user-1", "order_count": 15000}},
{"timestamp": "00:00:08Z", "level": "DEBUG", "event": "CALCULATING_USER_REPORT", "job_id": "job-1", "context": {"user_id": "vip-user-1"}},
{"timestamp": "00:01:30Z", "level": "DEBUG", "event": "CALCULATING_USER_REPORT_SUCCEEDED", "job_id": "job-1", "context": {"user_id": "vip-user-1"}},
{"timestamp": "00:01:30Z", "level": "DEBUG", "event": "PROCESSING_USER_END", "job_id": "job-1", "context": {"user_id": "vip-user-1"}},
{"timestamp": "00:01:30Z", "level": "DEBUG", "event": "PROCESSING_USER_START", "job_id": "job-1", "context": {"user_id": "vip-user-2"}},
// ... similar logs for user-2, user-3 ... then service crashes
]

Analysis Request:

1. Based on the timestamps, identify the most time-consuming operation within the loop.
2. What is the likely cause of the high CPU usage?
3. Propose a Go code-level optimization.

Step 4: Reason and Hypothesize (Reason & Hypothesize)

AI receives the logs and quickly provides its analysis.

AI (Go Performance Expert) Response:

1. Time-Consuming Operation: The CALCULATING_USER_REPORT step is the clear bottleneck. For vip-user-1, it took 82 seconds (from 00:00:08 to 00:01:30). The database query (FETCHING_USER_ORDERS) itself was relatively fast (6 seconds).

2. Likely Cause of High CPU: The logs show that for a single user (vip-user-1), the system fetches 15,000 orders and then spends a very long time in a "calculating" phase. This strongly suggests that the application is loading all 15,000 order objects into memory and then performing some CPU-intensive aggregation or calculation in a tight loop within the Go application itself. This in-memory processing of a large dataset is a classic cause of high CPU and memory pressure.

3. Proposed Optimization: The calculation logic should be pushed down into the database layer. Instead of fetching all orders and calculating in Go, we should use a single, optimized SQL query with aggregate functions (SUM, AVG, COUNT) and a GROUP BY clause to let the database do the heavy lifting. The application should only fetch the final, aggregated report data, which would be a single row per user.

You review AI's reasoning. This hypothesis is very reasonable! Pulling large amounts of data into application-layer memory for computation, rather than leveraging the database's computing power, is a classic performance anti-pattern. You decide to adopt this hypothesis.

Step 5: Verify and Fix (Verify & Fix)

• Verify: You do not need to wait until midnight. You find the code that handles this Job, point it at a test database containing massive order data in your local development environment, and manually trigger the Job. You also open a profiling tool. Sure enough, the profiler shows that 100% of CPU time is consumed in a massive for loop that is iterating over the huge number of order objects fetched from the database. The hypothesis is verified!
• Fix: You say to AI: "Your hypothesis was correct. Here is the problematic Go function. Please help me refactor it using the SQL aggregation approach you suggested." AI generates the new, efficient Go code and the corresponding SQL query for you.
• Regression test: You run the test again with the refactored code. The logic that used to take 82 seconds to process a single user now takes less than a second (illustrative figures for this teaching composite). CPU usage barely fluctuates during the entire process.
• Update logs: In the new code, you add a DEBUG log to the aggregation SQL query, recording the time it takes to execute. This way, if this query ever slows down in the future, it can be immediately detected.

The next midnight, you sleep soundly. When you wake up in the morning, the monitoring charts are perfectly calm. A "ghost" bug that had plagued the team for weeks (this case is a teaching composite) has been completely and elegantly resolved through a clear, human-machine collaborative, real-data-driven process.

This is the ultimate victory of "telemetry-driven development." It elevates your collaboration with AI from the shallow level of "code writing" to the deep level of "system diagnosis." You are no longer just "architects." You have also become "doctors," jointly safeguarding the health and stability of the software life you have created.