Pilot: one-shot vs plan-execute-verify on seeded-bug tasks
This is a small pilot (N=1 per task per condition, 4 tasks), run 2026-07-07 with haiku-class subagents under identical conditions. It precedes and does not replace the pre-registered N=10 benchmark. Raw data and per-run diffs are downloadable below.
Hypothesis
A plan-execute-verify loop will complete more of a seeded-bug task set than one-shot fixing with the same model, at higher cost, with a lower false-success rate.
Method
| Tasks | 4 small JavaScript modules, each with 1-2 seeded logic defects and a node --test suite confirmed failing at baseline (0/3, 1/3, 2/3, 4/6 tests passing respectively). |
|---|---|
| Conditions | Condition A (one-shot): read files, make one fix, no command execution or verification permitted. Condition B (PEV): plan with an explicit success criterion, fix, run the tests, verify observed output before declaring done, max 3 attempts. Same model class, same tool access otherwise. |
| Runs | N=1 per task per condition (8 runs total). A pilot, not a powered study. |
| Scoring | Ground truth: the auditor ran node --test in every run directory after agents finished. Binary completion, false-success rate (claimed success while tests fail), token and wall-clock telemetry. |
| Pinned | Task corpus, per-run diffs, and full results JSON published in the raw-data archive. Grading commands included. |
Results
| Measure | One-shot (A) | Plan-execute-verify (B) |
|---|---|---|
| Logic grading (harness-normalized) | 4/4 tasks fixed | 4/4 tasks fixed |
| Completed in the as-run environment | 0/4 | 4/4 |
| Claimed success | 4/4 | 4/4 |
| False-success rate | 4/4 | 0/4 |
| Avg tokens per run | 20,878 | 23,057 (+10%) |
| Avg wall time per run | 8.2s | 32.4s (~4x) |
The designed comparison tied: every agent in both conditions produced a correct logic fix, so these tasks were too easy to separate the conditions on bug-fixing skill. The environment separated them instead. The task corpus shipped with an unintended defect (a missing package.json making the ESM test suites unrunnable) that the author did not know about. All four one-shot agents, forbidden from verifying, confidently and wrongly claimed their tests would pass: a 100% false-success rate in the environment as it actually existed. All four PEV agents hit the failure on their first verification run, diagnosed it, repaired the environment unprompted, and reported truthfully observed passes. The pilot's honest headline is not that verification fixes bugs better; it is that verification catches the failure nobody knew was there, and that one-shot confidence is uncorrelated with as-run reality.
Raw data
Limitations
N=1 per cell and 4 tasks: indicative, not definitive. Tasks were author-constructed and small; real bug-fix distributions are harder and messier. Single model class; no claim of generalization across models. The environmental defect was accidental: a lucky, honest accident that demonstrates the mechanism, but a designed environment-perturbation study should replicate it deliberately. The pre-registered N=10 benchmark remains the definitive test.