Agent Benchmark Report Template
The reporting skeleton for a reproducible agent benchmark: hypothesis, pinned method, results, limitations, raw data. Copy it below, fill the bracketed fields, delete what you don't need.
# Benchmark: [title]
Status: [pre-registered / results published] · Date: [date]
## Hypothesis
[One falsifiable sentence, written before any runs.]
## Method
- Tasks: [N tasks, selection criteria, corpus link or hash]
- Conditions: [A vs B, everything else held equal]
- Runs: [N per task per condition]
- Model/tools pinned: [exact versions]
- Scoring: [binary criteria + rubric, written before runs]
## Results
| Condition | Completion | False-success rate | Cost/task | Notes |
|-----------|-----------|--------------------|-----------|-------|
| A | [x/N] | [x/N] | [tokens/time] | |
| B | [x/N] | [x/N] | [tokens/time] | |
## Limitations (mandatory, honest)
- [Sample size, task selection bias, model nondeterminism, version drift, author effects]
## Raw data
- [Link to transcripts/outputs, redacted as needed]
## Implications
[What this does and does not support. Resist the urge to generalize past the data.]
Field notes
| Field | Why it matters |
|---|---|
| Hypothesis before runs | Pre-registration is what separates measurement from marketing. The hypothesis and rubric predate the data or the report is a story. |
| Limitations | Mandatory and specific. Reports without limitations sections are advertisements. |