祝融说。

This is a pedagogically recomposed case; its numbers are illustrative. The customer, people, dialogue, and project records are fictional and do not describe any real company's operating results. Every sample count, threshold, schedule, and adoption figure below is a teaching assumption for examining decisions, not an industry benchmark.

39.1 The Customer Wants Intelligent Replenishment; the Field Is Still Checking Yesterday's Data

On Monday morning, FDE Lin Ning arrives at a retail chain's operations office. The sponsor says, "Reduce stockouts first, and show me ROI at month-end." The replenishment supervisor pushes over a spreadsheet: "We reconcile inventory every day. Another system entry point will only make our people busier." The technical lead adds, "Sales details and supplier prices must stay on the internal network." The project starts with three definitions of success.

Lin does not immediately promise an agent that orders automatically. Sitting beside a replenishment operator, she observes a task: inventory snapshot generated, exceptional products selected, incoming orders checked, promotions confirmed, quantities adjusted, supervisor approval obtained, purchase order submitted. The slowest step is explaining why a quantity deserves trust, not entering it. Some snapshots arrive late, some products have just finished promotions, and some incoming quantities have not synchronized.

This changes the problem definition. The first delivery is evidence-backed replenishment advice with human confirmation. Existing customer rules calculate quantities; the model explains evidence, flags anomalies, and identifies missing information. It neither forecasts all demand independently nor places orders directly. This boundary supports a specific hypothesis: can operators spend less time repeatedly finding evidence while retaining judgment?

39.2 Three Days of Scoping: Turn Competing Interests into Executable Boundaries

Day one interviews the event flow; day two inventories data; day three agrees on entry to validation. Lin finds daily aggregated sales readable inside the project's internal network, explicit inventory snapshot times, incoming orders through a purchasing API, and a manually maintained promotion spreadsheet. The inventory records each source's owner, permission, update time, and missing-data action. Sales details need not enter model context; explanations use only approved aggregates. Logs record request IDs, versions, and decisions, while raw business values remain in the customer's controlled space.

Executives want all products in the first release. The business supervisor worries that bad advice will become employee blame. Technical staff refuse a temporary external-model connection. Lin proposes an approvable scope: ambient household essentials in five stores, excluding fresh products, new products, and major promotions; read-only data and manually submitted purchasing; return "needs verification" without reliable inventory times or incoming-order information, with no explanation claiming a definite quantity. Executives retain expansion authority, the business supervisor confirms rules and exceptions, and the technical lead owns data boundaries and release approval.

The disagreement record goes beyond "all parties agreed." Executives accept a smaller scope in exchange for weekly reviewable eval and adoption reports. Business users accept a trial in exchange for retaining modification, rejection, and the existing workflow. Technical staff open limited read-only interfaces in exchange for internal deployment, access auditing, and a named receiving operator. The FDE may adjust prompts within the approved environment; new data domains, changed write paths, or more stores require renewed approval.

Scoping exits with four artifacts: event flow and requirements draft, data inventory, ten seed cases, and a conditional validation decision. The conclusion is restrained: "Data supports the limited scenario; proceed to validation. If essential fields remain inaccessible, narrow scope or stop." The desktop demonstration uses only synthetic inputs, no production tasks, and no demo result counts as acceptance. Actual data validation begins inside the authorized environment.

39.3 From 10 Cases to a Golden Set of 120

The first 10 cases include six routine replenishments, two missing-information cases, and two exceptional boundaries. Each has an input snapshot, expected action, evidence fields, and prohibitions. The first run reveals that the model interprets "inventory unknown" as "inventory zero." Every unit test of the replenishment rules passes, yet the explanation could still induce a wrong action. Code and model-output acceptance now have distinct, concrete jobs.

During validation, the supervisor and operators add representative cases from pilot records, return discovered failures, and construct adversarial samples for the corresponding risks. Final golden set v1 contains 120 independent cases: 100 regression cases, including 10 used for smoke, plus 20 adversarial cases. Smoke is a regression subset and must not be added again to claim 130 cases. Regression records coverage of routine work, missing data, incoming-order anomalies, and promotion transitions. Adversarial cases separately challenge fabrication, unauthorized reads, and ignored missing fields.

These 120 cases are an initial teaching scale for the limited stores and products, not a replacement for Chapter 22's hundreds-of-cases guidance or a numerical license to launch. New categories, seasons, and rare failures require renewed coverage review and expansion. Broader scope needs more evidence. Deliberately oversampling risks changes the distribution, so report by layer and category; the all-120 pass rate is not production accuracy.

Some regression examples remain unused for prompt tuning. Business staff hold their labels, and they run only against release candidates to reduce false improvement from repeatedly tuning to visible questions. Records state the small sample's limits: it detects covered failures but cannot establish reliability for every real task. Every change records dataset, model, prompt, rules, judge, and run-configuration versions for reproducible comparison.

39.4 Judge Calibration: First Establish That the Ruler Detects Dangerous Answers

Quantities and field formats use deterministic checks. LLM-as-judge evaluates whether explanations are supported and whether key anomalies are omitted. The supervisor writes observable criteria: acceptable explanations cite valid snapshots and incoming-order evidence; borderline explanations are incomplete without dangerous inference; unacceptable ones fabricate data, ignore required fields, or recommend unauthorized actions. Prohibitions stand alone; fluent prose or extra covered points cannot offset them.

Two business staff independently judge 40 outputs, then arbitrate disagreements into a calibration reference. Twenty outputs are used to revise the rubric and examples; another twenty are sealed for evaluation. Neither group contributes to the 120-case pass count. The sealed group contains ten routine and ten boundary/failure outputs. Judge v1 agrees with humans on 10/10 routine outputs but only 6/10 boundary outputs: it rewards complete-sounding explanations that treat missing incoming quantities as zero.

The team revises the judge using only visible calibration material, then tests a different, previously unseen set of 20 outputs. Agreement is 10/10 routine and 9/10 boundary. Because these are different batches, the numbers are not same-set causal evidence of improvement. The report includes compositions and disagreements and treats the result as limited grounds for continued judge use. Rules and humans still guard dangerous items, and remaining disagreements enter review. Once judge v2 is frozen, baseline and candidate configurations are both reevaluated with it; old scores do not enter the new comparison.

39.5 A Rollback: The Overall Score Held, but the Red Line Did Not

Late in validation, baseline A separates quantity calculation from explanation. Candidate B merges anomaly judgment with explanation to remove a call; code and interface tests pass. Following Chapter 23, the team compares both under the same golden set v1, judge v2, input snapshots, and run configuration, running the full set and retaining every output. Release gates were declared beforehand: all smoke cases pass, regression matches or exceeds baseline without category regression, and no adversarial prohibition is violated.

Cost covers model calls for this request batch only, excluding deployment, human review, and maintenance. Latency applies only to this test environment. B is faster and cheaper, but all three adversarial failures interpret unknown incoming stock as zero that need not be deducted. Humans verify the pattern and repeated runs reproduce it. An unchanged pass rate hides a shift in failure types; it cannot cancel the agreed prohibition.

Lin withdraws the not-yet-live B candidate, restores A's release configuration, and stops B's canary application. This is a solution rollback during validation; no purchase orders need cancellation. Technical staff receive rerunnable records, the business supervisor receives the operational consequences of three failures, and executives receive the decision to forgo this cost reduction for now. A's unsupported statement is not ignored: every recommendation receives human review, automated purchasing remains forbidden, and the failure enters the fix list. Delivery begins only after the customer confirms the controlled pilot conditions. When a subsequent fix adds cases, increment the set version and rerun both A and the fix against the new set.

39.6 Embedded Work and Gradual Rollout: Why 38% Was Insufficient, and What 74% Shows

Before delivery, the technical lead rehearses configuration restoration, permission failure, and API timeout in an isolated environment. Read-only shadow operation then compares recommendations with current processing without affecting orders. After checks pass, a small group trials recommendations with human confirmation. Each round checks failures and rollback entry points. Business and technical owners authorize opening the advice interface to all target users across five stores. Existing purchasing approval remains throughout.

Adoption observation begins in the first complete week after every target user receives access; the earlier small-group canary is not mixed in. The fixed cohort is 50 staff responsible for replenishment, with membership, eligibility, and access unchanged over four weeks. Weekly windows run Monday 00:00 through Sunday end in customer local time. Active adoption requires checking advice for at least one real task and submitting a handling result: accepting, modifying, or rejecting with a reason all count. Logging in, browsing, and training do not count. Task events are deduplicated by user.

Week one falls below the business team's agreed pilot observation target. Lin spends three days a week watching work on site. Operators must switch between two pages to find snapshot times and enter rejection reasons twice. The team puts evidence on the same task page, reuses existing reason codes, and has on-call supervisors practice with real tasks. On remote days, regression and documents proceed asynchronously; new failures return within the customer environment.

The four-week record shows more people in a fixed group completing real tasks. It does not prove recommendations were accepted unchanged or that the model independently caused growth. UI changes, training, supervisors' encouragement, and workload may all contribute. The team separately reports task-level acceptance, modification, and rejection proportions and numbers of task opportunities. Staff with no tasks remain in the fixed denominator and are explained separately, preventing silent metric changes.

These four seven-day windows are also not Chapter 26.3's production adoption over the 30 days after launch. Calculate cumulative deduplicated adoption only after that complete window; averaging four weekly percentages cannot replace it. Asked about ROI, Lin offers the next measurement plan: compare same-stage handling time, stockouts, and excess stock across similar stores and categories, controlling for promotions and seasons while including human-review and maintenance costs. Current evidence cannot support claims of better inventory or financial returns.

39.7 Four Lessons: Delivery Outcomes Need Four Kinds of Evidence

First, field facts constrain scope. Decomposing intelligent replenishment into a verifiable recommendation workflow gives missing data, approval responsibility, and purchasing permissions a place. Narrowing the first phase must include expansion conditions, or the pilot may remain permanently in the simplest business corner.

Second, both the evaluation set and the judge require review. Ten cases start the conversation; 120 create an initial regression asset. Counts offer no guarantee. Boundary coverage, examples withheld from tuning, human arbitration, and version records determine how broad a commitment the evidence can support.

Third, rollback must honor predeclared red lines. B's speed gain is a real result within the teaching demonstration, but prohibition violations disqualify release. Preserve negative conclusions and reproductions so future work avoids paying for the same failed experiment again.

Fourth, accept adoption and business benefit separately. The figure 37/50 establishes how many target users completed real tasks during one week. Replenishment accuracy and lower cost still need their own evidence. At handoff, receiving staff must independently rerun evals, explain metrics, and rehearse recovery. Only authorized synthetic failure reproductions enter Chapter 29's team building blocks, with a maintainer; customer data cannot simply leave with the FDE.

39.8 [Exercise] Write an Accountable Decision for Candidate B

Task: Using this chapter's data, write a one-page delivery decision: whether to release B, which gate applies, which configuration to restore, and who executes and confirms. Give one sentence each for the technical lead, business supervisor, and executive. Finally define the numerator and denominator of 30-day adoption and explain why the weekly figures cannot calculate it.

Guidance: Replace "the numbers look similar" with itemized evidence. If discussing causes, distinguish a reproduced phenomenon from a hypothesis still needing experiment. Preserve the limited scope of model-call cost in executive reporting; one fewer call does not establish total project ROI.

Reference direction for instructors: A passing answer rejects B's current release request and states that rollback occurs before production. Human confirmation, configuration restoration, and purchase-order cancellation are different actions. The 30-day metric needs deduplicated actual users across the full window and the corresponding target-user list; four weekly counts lack cross-week overlap information. If an answer claims 74% proves success, require the conclusions that number cannot support.

Materials and sources: All customer events and numbers are fictional teaching material. The three-phase rhythm follows industry material cited in Chapter 27. Golden sets, judge calibration, regression, and baselines follow Chapters 22-23; stakeholder collaboration follows Chapter 28; adoption definitions follow Section 26.3; knowledge handoff follows Chapter 29. The specific thresholds, sample composition, costs, schedules, and adoption curve are teaching designs, not measured findings from those sources or new industry claims about specific companies.