Building with agents 0/15

Lesson 7 of 8

Verify a generated change before claiming success

Take a bug report through reproduction, a focused repair and an evidence-backed handoff.

906 words · 5 min read

Reproduce the behavior you need to change

Verification means checking a result against a requirement using evidence appropriate to the claim. A source edit shows what you changed. A test result shows behavior on particular inputs. A deployment receipt identifies a released version, and a user-flow check observes behavior in that environment. Each answers a different question.

Suppose a teammate reports, “A subtotal of 50 still charges shipping, although free shipping starts at 50.” A bug is behavior that violates the agreed requirement. First reproduce the reported input and record the expected result. Otherwise, a generated patch may improve some other behavior while leaving this defect intact.

Text
Requirement: free shipping for subtotal >= 50.
Input domain: valid nonnegative whole currency amounts.

Current JavaScript function:
function freeShipping(subtotal) {
  return subtotal > 50;
}

Reported input: 50
Expected result: true
Original result: false
A complete local reproduction of the shipping defect

The > comparison excludes equality, while >= includes it. The failing input is on the boundary, the value where the required result changes from ineligible to eligible. A subtotal of 60 exercises the eligible side without exercising equality, so its success cannot answer this bug report.

Text
Fix the freeShipping function so the requirement includes 50.
First run a test showing the current result at 50.
Keep that test, change the implementation, and rerun it.
Also check 49 and 60 to preserve nearby behavior.
Show the diff and actual test results.
Report local verification separately from deployment.
A repair request with a checkable result

A regression test preserves an expected behavior so later changes can be checked against it. For this defect, the test must fail on the original function and pass on the repair. If it already passes before the edit, inspect whether it actually calls the affected function with 50. A test that never reaches the bug cannot demonstrate its repair.

Inspect the repair and execute nearby cases

A diff displays the lines removed and added by a change. A minus sign marks removed text, and a plus sign marks added text. Review it before concluding that a task is complete, because generated edits can include unrelated behavior. The requirement here calls for one comparison to change.

Diff
- return subtotal > 50;
+ return subtotal >= 50;
The focused implementation change

The repaired function gives the expected boundary result. Check the neighboring behavior too: 49 remains below the threshold and 60 remains above it. For the chosen whole-number domain, 49 is the nearest lower value. If the real function accepted cents or decimal amounts, choose cases in that domain and establish how rounding works before extending this conclusion.

You can reproduce these checks without a test framework. If you have Node.js installed, save the following as shipping-check.js and run node shipping-check.js. Each pair contains an input and its expected result. The loop calls the function and throws an error on a mismatch, which ends the run with a failure. To observe the original defect, first use > in the function; then restore >= and rerun the same checks.

JavaScript
function freeShipping(subtotal) {
  return subtotal >= 50;
}

const cases = [[49, false], [50, true], [60, true]];
for (const [subtotal, expected] of cases) {
  const actual = freeShipping(subtotal);
  if (actual !== expected) {
    throw new Error(`${subtotal}: expected ${expected}, received ${actual}`);
  }
  console.log(`${subtotal}: ${actual}, correct`);
}
A complete local check you can run and modify
Text
Case          Expected   Original   Proposed
Subtotal 49   false      false      false
Subtotal 50   true       false      true
Subtotal 60   true       true       true
The executed comparison in the local teaching fixture

The boundary case distinguishes the original and repaired functions. The other cases check behavior that should remain unchanged. A test suite is a collection of tests; its size is less useful here than whether it reaches the behavior at issue. Hundreds of unrelated successful tests would not replace the missing check at 50.

A failure can also expose a bad proposed repair. Suppose an access rule currently lets only an item’s owner read it, and the request adds previews for public items. A proposal of isOwner || preview allows any preview, including a private one. Testing only the owner’s successful read misses that change.

For that second task, write expected outcomes before judging the patch: the owner may read a private item; a non-owner may preview a public item; a non-owner must be denied a private preview. The last case checks a prohibited action, sometimes called a negative test. A failure there is a reason to reject the proposal and include the public-item condition in the next repair. It is useful verification even though the generated change was wrong.

Write a handoff that matches the evidence

After the shipping checks succeed, produce a result someone can review and use. Include the affected function, the observed behavior and the checked scope. The following is an example handoff for the local fixture, rather than a claim about a real deployed checkout.

Text
Updated freeShipping to include subtotal 50.
The boundary test fails on the original comparison and passes on the repair.
All three local cases pass: 49 -> false, 50 -> true, 60 -> true.
The diff changes only the eligibility comparison and its regression test.
Deployment and the checkout flow remain unverified.
A handoff tied to specific local evidence

If a selected test fails, keep its input and actual output, identify the violated requirement, and return to the repair. After another edit, rerun the failing test and affected nearby cases. Removing the failing assertion would remove the evidence of the defect while leaving the defect possible.

If a command fails before running any tests, report that execution problem. “The runner could not load shipping.test.js” establishes a missing check, rather than a failing shipping result. Preserving the distinction helps the next person fix the correct problem.

To verify a released checkout later, identify the deployed version containing the patch and exercise the shipping calculation through the actual checkout flow. A local unit test calls a small function directly; it can miss a page calling a different function or sending a different subtotal. Anthropic’s agent-evaluation guidance makes the same distinction between an assistant’s record of actions and the resulting state in the environment.

Choose evidence that reaches the affected surface. For a layout change, inspect the rendered page at the relevant screen sizes. For an access change, use the relevant user roles and public/private states. Keep the result of each check attached to the version it exercised, since a later edit can invalidate earlier results.

The lab executes the original and proposed shipping comparisons for the cases you select. Run the boundary and both nearby cases before choosing the conclusion. The assessment uses the preview-access proposal, where successful verification requires noticing the prohibited private preview.

Try the skill

Contract: free shipping applies to a subtotal of 50 or more. Inspect the JavaScript proposal, select the checks and run them. Choose a conclusion that matches the evidence.

JavaScript
// Before
return subtotal > 50;

// Proposed change
return subtotal >= 50;
Select checks to execute
Your conclusion

Primary references