祝融说。

Debugging an autonomous agent is different from debugging a regular program—LLM outputs are non-reproducible, and the causal chain of an error may span multiple tool calls.

Pattern Layer

15.1 Unique Challenges of Agent Observability

Traditional software observability (logs, metrics, traces) reveals three shortcomings when applied to agent systems:

First, non-deterministic output. A regular program produces the same output for the same input (deterministic). An agent system may produce different output for the same input—LLM non-determinism makes "reproducing a bug" difficult. When the user says "refactor module A," the agent's first attempt may produce a completely different solution than its second attempt. You cannot verify a fix by simply "running it again."

Second, the causal chain spans multiple tool calls. A single agent decision may involve 5 tool calls, 3 LLM round trips, and 1 sub-agent invocation. If the final output is wrong, the root cause may lie in the first step—but the reasoning from that first step has already been overwritten by subsequent context. The traditional log pattern of "search by timestamp" is inefficient when faced with causal chains spanning multiple steps.

Third, LLM output is not parseable. Structured events in traditional logs ("user login successful," "database query returned 0 rows") can be precisely parsed and filtered. LLM reasoning tokens are natural language—they cannot be structurally parsed by a program. You cannot grep for "error cause" to find why an agent decision went wrong.

15.2 Three Pillars of Observability

To address these challenges, the CodeCoder observability system is designed around three pillars:

Structured event stream. All agent events (NewToken, ToolStarted, ToolFinished, MilestoneDone, StatusUpdate) are emitted as structured JSON events. Each event includes a timestamp, event type, and relevant context. Structured events can be programmatically parsed, filtered, and sampled.

Real-time observability. In headless mode, the event stream is simultaneously written to stderr and .ccd.bg.ndjson. In interactive mode, the event stream is broadcast to all connected clients via the daemon's event channel. There is no need to wait for a run to complete to see progress.

Post-hoc traceability. Session files preserve the complete event history. The BgObserver-written ndjson file retains the full event stream from the last truncation to the current run. For post-mortem analysis, there is no need to re-run the agent—reading the ndjson file suffices.

Case Layer

15.3 BgObserver + bg_ledger

BgObserver is the observability component in headless mode:

struct BgObserver {
    ndjson_writer: BufWriter<File>,
    events_seen: usize,
    start_time: Instant,
}

impl BgObserver {
    fn new(project_root: &Path) -> Result<Self> {
        let path = project_root.join(".ccd.bg.ndjson");
        // Truncate at start of each round
        let file = fs::OpenOptions::new()
            .write(true)
            .truncate(true)
            .create(true)
            .open(&path)?;
        Ok(Self {
            ndjson_writer: BufWriter::new(file),
            events_seen: 0,
            start_time: Instant::now(),
        })
    }

    fn observe(&mut self, event: &BgEvent) {
        // Write to ndjson
        let json = serde_json::to_string(event)?;
        writeln!(self.ndjson_writer, "{}", json)?;
        self.ndjson_writer.flush()?;
        // Also write to stderr
        eprintln!("{}", json);
        self.events_seen += 1;
    }
}

bg_ledger is an extension of BgObserver—it records the duration and result of each milestone:

struct BgLedger {
    milestone_times: Vec<MilestoneRecord>,
    tool_counts: HashMap<String, usize>,
    total_tokens: usize,
}

struct MilestoneRecord {
    name: String,
    started_at: Instant,
    completed_at: Option<Instant>,
    status: MilestoneStatus,
    attempts: usize,
}

15.4 Accountability Chain

The Accountability Chain is an advanced feature in the CodeCoder observability system—it links an agent's decisions to the concrete outputs they produce.

When the agent makes a decision (e.g., "modify the interface of file A"), the Accountability Chain records:

• The trigger condition for the decision (user input)
• A summary of the decision's reasoning process (key points extracted from reasoning)
• The sequence of tool calls that executed the decision
• The execution result (diff of file modifications)

These records help answer, during post-mortem analysis, "why did the agent make this change"—rather than just "what change did the agent make."

struct AccountabilityEntry {
    timestamp: Instant,
    trigger: String,          // Input that triggered the decision
    rationale: String,        // Reasoning process summary
    actions: Vec<ToolCall>,   // Executed tool calls
    outcome: String,          // Execution result summary
    diff: Option<String>,     // File modification diff (if any)
}

15.5 Debugging Methodology

The methodology for debugging autonomous agents can be summarized in three steps:

Step 1: Isolate LLM output. Replace the actual LLM provider with StubClient or ScriptedProvider. StubClient returns fixed responses; ScriptedProvider plays back from a pre-recorded sequence of responses. This makes the agent's behavior deterministic during debugging—each run produces the same output.

Step 2: Deterministic replay. Read the event sequence from a Session file or ndjson file and replay it without connecting to an LLM provider. During replay, the agent's input comes from recorded events rather than real-time LLM output. This allows fast identification of "after which event did the agent start exhibiting abnormal behavior."

fn replay(session: &Session, context: &Context) -> Result<()> {
    for event in &session.events {
        match event {
            Event::NewToken(_) => {}  // Ignore token output
            Event::ToolStarted { tool, args } => {
                // Check if tool call is reasonable
                validate_tool_call(tool, args)?;
            }
            Event::ToolFinished { tool, result } => {
                // Validate tool result
                validate_tool_result(tool, result)?;
            }
            Event::MilestoneDone { name, .. } => {
                // Validate milestone completion conditions
                validate_milestone(name)?;
            }
        }
    }
    Ok(())
}

Step 3: Structured event tracing. Use event filtering tools (such as jq) to extract specific event types from the ndjson file for analysis:

# Extract all tool call events
jq 'select(.event == "ToolStarted")' .ccd.bg.ndjson

# Extract all failed tool calls
jq 'select(.event == "ToolFinished" and .result.status == "error")' .ccd.bg.ndjson

# Extract all milestone completion events, sorted by time
jq 'select(.event == "MilestoneDone") | {name, timestamp}' .ccd.bg.ndjson

ADR Deep Dive

Motivation for Introducing BgObserver and bg_ledger

ADR 0039 documents the introduction of BgObserver and bg_ledger.

Before BgObserver existed, the output of headless mode consisted only of an exit code and error messages on stderr. If a headless run failed (exit code 2 or 3), the user had to re-run to see "what went wrong." Re-running could produce different results (LLM non-determinism), making "reproducing the bug" difficult.

The introduction of BgObserver transformed the headless run process from a "black box" to a "transparent box." Each event is written to an ndjson file, allowing users to tail -f for real-time observation or perform offline analysis after the run completes. The ndjson format (one JSON event per line) ensures the file can be stream-processed—there is no need to wait for the run to finish before beginning analysis.

BgLedger builds on BgObserver by adding milestone-level aggregate statistics. Rather than recording every event, it records each milestone's duration, tool call count, and retry count. This makes root cause analysis for "why is this run slow" feasible—if a milestone's retry count is far above average, you can target that milestone's detailed events for inspection.

The Accountability Chain was added later. Its motivation was: when the agent makes an incorrect modification (e.g., deleting code it should not have), existing logging can only tell you "the agent deleted code," not "why the agent thought this code should be deleted." The Accountability Chain associates the reasoning process summary of a decision with its execution outcome, enabling post-mortem review to reveal "motivation" rather than merely "action."

Next chapter: CodeCoder's testing strategy—a three-tier test pyramid, isolation testing, and behavioral validation.