Reproduce the behavior you need to change
Verification means checking a result against a requirement using evidence appropriate to the claim. A source edit shows what you changed. A test result shows behavior on particular inputs. A deployment receipt identifies a released version, and a user-flow check observes behavior in that environment. Each answers a different question.
Suppose a teammate reports, “A subtotal of 50 still charges shipping, although free shipping starts at 50.” A bug is behavior that violates the agreed requirement. First reproduce the reported input and record the expected result. Otherwise, a generated patch may improve some other behavior while leaving this defect intact.
Requirement: free shipping for subtotal >= 50.
Input domain: valid nonnegative whole currency amounts.
Current JavaScript function:
function freeShipping(subtotal) {
return subtotal > 50;
}
Reported input: 50
Expected result: true
Original result: falseThe > comparison excludes equality, while >= includes it. The failing input is on the boundary, the value where the required result changes from ineligible to eligible. A subtotal of 60 exercises the eligible side without exercising equality, so its success cannot answer this bug report.
Fix the freeShipping function so the requirement includes 50.
First run a test showing the current result at 50.
Keep that test, change the implementation, and rerun it.
Also check 49 and 60 to preserve nearby behavior.
Show the diff and actual test results.
Report local verification separately from deployment.A regression test preserves an expected behavior so later changes can be checked against it. For this defect, the test must fail on the original function and pass on the repair. If it already passes before the edit, inspect whether it actually calls the affected function with 50. A test that never reaches the bug cannot demonstrate its repair.
Inspect the repair and execute nearby cases
A diff displays the lines removed and added by a change. A minus sign marks removed text, and a plus sign marks added text. Review it before concluding that a task is complete, because generated edits can include unrelated behavior. The requirement here calls for one comparison to change.
- return subtotal > 50;
+ return subtotal >= 50;The repaired function gives the expected boundary result. Check the neighboring behavior too: 49 remains below the threshold and 60 remains above it. For the chosen whole-number domain, 49 is the nearest lower value. If the real function accepted cents or decimal amounts, choose cases in that domain and establish how rounding works before extending this conclusion.
You can reproduce these checks without a test framework. If you have Node.js installed, save the following as shipping-check.js and run node shipping-check.js. Each pair contains an input and its expected result. The loop calls the function and throws an error on a mismatch, which ends the run with a failure. To observe the original defect, first use > in the function; then restore >= and rerun the same checks.
function freeShipping(subtotal) {
return subtotal >= 50;
}
const cases = [[49, false], [50, true], [60, true]];
for (const [subtotal, expected] of cases) {
const actual = freeShipping(subtotal);
if (actual !== expected) {
throw new Error(`${subtotal}: expected ${expected}, received ${actual}`);
}
console.log(`${subtotal}: ${actual}, correct`);
}Case Expected Original Proposed
Subtotal 49 false false false
Subtotal 50 true false true
Subtotal 60 true true trueThe boundary case distinguishes the original and repaired functions. The other cases check behavior that should remain unchanged. A test suite is a collection of tests; its size is less useful here than whether it reaches the behavior at issue. Hundreds of unrelated successful tests would not replace the missing check at 50.
A failure can also expose a bad proposed repair. Suppose an access rule currently lets only an item’s owner read it, and the request adds previews for public items. A proposal of isOwner || preview allows any preview, including a private one. Testing only the owner’s successful read misses that change.
For that second task, write expected outcomes before judging the patch: the owner may read a private item; a non-owner may preview a public item; a non-owner must be denied a private preview. The last case checks a prohibited action, sometimes called a negative test. A failure there is a reason to reject the proposal and include the public-item condition in the next repair. It is useful verification even though the generated change was wrong.
Write a handoff that matches the evidence
After the shipping checks succeed, produce a result someone can review and use. Include the affected function, the observed behavior and the checked scope. The following is an example handoff for the local fixture, rather than a claim about a real deployed checkout.
Updated freeShipping to include subtotal 50.
The boundary test fails on the original comparison and passes on the repair.
All three local cases pass: 49 -> false, 50 -> true, 60 -> true.
The diff changes only the eligibility comparison and its regression test.
Deployment and the checkout flow remain unverified.If a selected test fails, keep its input and actual output, identify the violated requirement, and return to the repair. After another edit, rerun the failing test and affected nearby cases. Removing the failing assertion would remove the evidence of the defect while leaving the defect possible.
If a command fails before running any tests, report that execution problem. “The runner could not load shipping.test.js” establishes a missing check, rather than a failing shipping result. Preserving the distinction helps the next person fix the correct problem.
To verify a released checkout later, identify the deployed version containing the patch and exercise the shipping calculation through the actual checkout flow. A local unit test calls a small function directly; it can miss a page calling a different function or sending a different subtotal. Anthropic’s agent-evaluation guidance makes the same distinction between an assistant’s record of actions and the resulting state in the environment.
Choose evidence that reaches the affected surface. For a layout change, inspect the rendered page at the relevant screen sizes. For an access change, use the relevant user roles and public/private states. Keep the result of each check attached to the version it exercised, since a later edit can invalidate earlier results.
The lab executes the original and proposed shipping comparisons for the cases you select. Run the boundary and both nearby cases before choosing the conclusion. The assessment uses the preview-access proposal, where successful verification requires noticing the prohibited private preview.
Try the skill
Contract: free shipping applies to a subtotal of 50 or more. Inspect the JavaScript proposal, select the checks and run them. Choose a conclusion that matches the evidence.
// Before
return subtotal > 50;
// Proposed change
return subtotal >= 50;