One-shot prompting vs plan-execute-verify on bug-fix tasks
Protocol pre-registered 2026-07-07. Runs in progress. Results publish when all conditions complete; this page is immutable except for the results section, which is empty until then.
Hypothesis
A plan-execute-verify loop will complete more of a fixed bug-fix task set than one-shot prompting with the same model and tools, at higher cost per task, with a lower false-success rate.
Method
| Tasks | 10 bug-fix tasks drawn from real open-source issues with failing tests, selected before any runs and published in the task corpus repo. |
|---|---|
| Conditions | Condition A: single prompt containing the issue and repo context. Condition B: plan-execute-verify loop with per-step criteria. Same model, same tool access, same context budget. |
| Runs | N=10 runs per task per condition (100 runs per condition total). |
| Scoring | Binary completion (original failing test passes, no other tests broken), quality rubric 0-100 with anchors written before runs, false-success rate (agent claimed done, tests disagree), cost and wall-clock telemetry. |
| Pinned | Model version, tool versions, task corpus commit hash, and rubric are pinned and will be published with raw transcripts. |
Results
Empty until runs complete. This section is the only part of the page that will change, and the change will be logged in the changelog.
Limitations
Model nondeterminism at temperature; small task N; task-selection bias toward test-covered bugs; single-model design says nothing about generalization across models; the loop implementation is one of many possible PEV variants.