祝融说。

Testing an autonomous agent system—deterministic tests cover deterministic logic, isolation tests cover tool behavior, and real LLM tests validate end-to-end capabilities.

Pattern Layer

16.1 Three-Tier Test Pyramid

Traditional software testing pyramid: unit tests → integration tests → E2E tests.

The testing pyramid for agent systems needs adjustments specific to agents:

         ┌──────────┐
         │  Real LLM│  ← L3: Real LLM smoke test (gated, not run by default)
         │  Smoke    │
        ┌┴──────────┴┐
        │  Black-box │  ← L2: ScriptedProvider simulates LLM responses
        │  Behavior  │
        │  Validation│
       ┌┴───────────┴┐
       │  Offline     │  ← L1: StubClient, no LLM calls
       │  Unit Tests  │
       └─────────────┘

L1: Offline unit tests. No LLM calls involved. Tests cover tool execution logic, permission lookup chains, message serialization/deserialization, and work graph scheduling. Uses StubClient—an LLM client that returns fixed responses. L1 tests run by default (cargo test).

L2: Black-box behavioral validation. Uses ScriptedProvider—playback from a pre-recorded sequence of responses, simulating the LLM's conversational behavior. Tests whether the agent's behavior under given LLM responses meets expectations: correct tool call order, whether acceptance gates are triggered, and whether sub-agents are created as expected. L2 tests run by default but are gated with #[cfg(feature = "integration")].

L3: Real LLM smoke tests. Connects to a real LLM provider to test the agent's end-to-end capabilities. Requires setting CODECODER_API_KEY. L3 tests do not run by default (#[ignore]), requiring explicit invocation.

16.2 Special Challenges in Agent Testing

Agent system testing faces two special challenges:

Challenge 1: Non-deterministic output. The same prompt may produce different outputs at different times or with different model versions. The L2 ScriptedProvider solves this—by pre-recording LLM responses, tests can deterministically verify agent behavior.

Challenge 2: State dependency. Agent behavior depends on the current context state. Tests need precise control over the agent's starting state—including system prompt, session history, and work graph state. L2 tests solve this by resetting state before testing and injecting predefined context.

Case Layer

16.3 StubClient

StubClient is the LLM provider for L1 tests. It does not call a real API but returns fixed responses:

struct StubClient {
    response: String,
}

impl ProviderClient for StubClient {
    fn send(&self, _request: &Request) -> Result<Response> {
        Ok(Response {
            content: vec![ContentItem::Text(self.response.clone())],
            tool_calls: vec![],
        })
    }
}

Use cases: testing tool execution logic, permission lookup chains, and work graph scheduling—these do not depend on the specific output of the LLM; they only require the agent to call tools in the correct order.

#[test]
fn test_tool_permission_chain() {
    let client = StubClient::new("read src/mod.rs");
    let agent = AgentLoop::new(client, default_permissions());
    let result = agent.process_turn();
    assert!(result.is_ok());
    // Verify the agent called the read_file tool
    assert!(agent.tool_calls().contains("read_file"));
}

16.4 ScriptedProvider

ScriptedProvider is the LLM provider for L2 tests. It plays back from a pre-recorded sequence of responses:

struct ScriptedProvider {
    responses: VecDeque<Response>,
}

impl ScriptedProvider {
    fn from_file(path: &str) -> Result<Self> {
        let content = fs::read_to_string(path)?;
        let responses: Vec<Response> = serde_json::from_str(&content)?;
        Ok(Self {
            responses: VecDeque::from(responses),
        })
    }
}

impl ProviderClient for ScriptedProvider {
    fn send(&mut self, request: &Request) -> Result<Response> {
        // Verify the request meets expectations (optional)
        self.verify_request(request)?;
        // Return the next pre-recorded response
        self.responses.pop_front()
            .ok_or_else(|| "no more responses".into())
    }
}

Use cases: testing agent behavior under given LLM responses. For example, if the LLM returns two ToolCalls—the agent should execute them in sequence; if the LLM returns a NeedsFix signal—the review gate should trigger the needs_fix flow.

#[test]
fn test_two_tool_calls_in_sequence() {
    let provider = ScriptedProvider::from_file("tests/fixtures/two-tool-calls.json")?;
    let agent = AgentLoop::new(provider, default_config());
    let result = agent.process_turn();
    assert!(result.is_ok());
    // Verify the agent executed two tool calls in sequence
    assert_eq!(agent.executed_tools().len(), 2);
    assert_eq!(agent.executed_tools()[0].name(), "glob");
    assert_eq!(agent.executed_tools()[1].name(), "read_file");
}

16.5 L3 Real LLM Smoke Tests

L3 tests use a real LLM provider to validate the agent's end-to-end capabilities:

#[test]
#[ignore]  // Not run by default
fn test_end_to_end_refactoring() {
    // Set up real LLM provider
    let provider = OpenAIClient::from_env()?;
    let agent = AgentLoop::new(provider, full_config());

    // Send the task
    agent.process_message("Extract module A's interface into a standalone trait")?;

    // Verify the result
    let workgraph = agent.workgraph();
    assert!(workgraph.is_completed());
    assert!(workgraph.all_milestones_done());
    // Verify the file was modified
    assert!(Path::new("src/interface.rs").exists());
}

The gating strategy for L3 tests:

• Not run by default in CI (requires CODECODER_API_KEY)
• Marked as #[ignore], requiring explicit cargo test --include-ignored
• Use isolated test projects (tests/fixtures/), do not modify the main project code

16.6 Test Statistics

CodeCoder's test statistics at the time of writing:

Total tests: 481
L1 (offline unit tests): ~450
L2 (black-box behavioral validation): ~28
L3 (real LLM smoke tests): 3 (all #[ignore])

L1 tests cover core logic including tool execution, permission lookup chains, message serialization, work graph scheduling, and compaction. L2 tests cover behavior paths including the acceptance pipeline, sub-agent creation, and headless execution. L3 tests cover only the three most critical end-to-end paths (refactoring, review, headless execution).

16.7 Tests as Living Specifications

One of CodeCoder's design principles: tests are living specifications.

Traditionally, specification documents and tests are separate—specification documents describe "what the system should do," and tests verify "what the system actually does." When specifications and tests are inconsistent, the tests are usually the accurate ones (code does not lie).

CodeCoder's testing strategy goes further: test code directly reflects the system's behavioral contract. Permission tests describe "what operations require what level of permission." Acceptance tests describe "what conditions constitute milestone completion." These tests are not just verification tools—they are executable specifications of system behavior.

An example—permission key tests:

#[test]
fn test_permission_key_precision() {
    // Verify that run_command:git does not match git status && git push
    let key = PermissionKey::parse("run_command:git");
    assert!(key.matches("git status"));       // Simple command matches
    assert!(!key.matches("git status && git push")); // Compound command does not match (hashed keying)
}

This test simultaneously serves as: test code (verifies behavior), specification document (describes "compound command keying rules"), and contract (tells maintainers "which tests need updating when this rule changes").

ADR Deep Dive

Layered Evolution of the Test System

CodeCoder's test system evolved from a single tier to three tiers.

Initial phase (single tier): Only L1 tests existed, using StubClient. Tests covered tool logic and permission checks, but not agent behavior paths. Problem: changes in agent behavior paths (such as the LLM returning an unexpected ToolCall) would not be caught by tests.

Second phase (two-tier introduction): L2 tests (ScriptedProvider) were introduced. By recording LLM response sequences, the agent's behavior under given inputs could be verified. Tests for the acceptance pipeline, sub-agent creation, and headless execution were all added in this phase.

Third phase (L3 smoke test introduction): L3 tests (real LLM) were introduced. L2 tests verified the agent's behavior under given LLM responses, but "whether the LLM will generate the expected response in a real scenario" was not within L2's coverage. L3 tests use a real LLM to verify end-to-end paths—but due to cost and determinism considerations, L3 tests are not run by default.

The introduction sequence of the three-tier test system reflects CodeCoder's evolution path: first deterministic logic (L1), then behavior paths (L2), and finally end-to-end validation (L3). Each tier was introduced because problems that the previous tier could not cover were exposed in real usage.

—End of the Technical Volume—

Five parts across 16 chapters covering a complete system from philosophical foundations to engineering practice. If you have read through to here, you should now possess all the core judgment frameworks and engineering implementation details needed to design an autonomous agent system.

The appendix contains a glossary, project data sheet, and ADR index. Please refer to the appendix for a complete reference.