Changelog
Every methodology change, data correction, and substantive edit, in public.
First real benchmark results published: a pilot of one-shot vs plan-execute-verify (4 seeded-bug tasks, live agent runs, ground-truth graded). Logic fixing tied 4/4; the as-run environment separated the conditions completely: one-shot false-success rate 4/4, PEV 0/4, because verification surfaced and repaired an environmental defect unknown to the task author. Raw data downloadable on the report page.
Four benchmark protocols pre-registered: superpowers vs baseline, auto-trigger vs slash-command firing, memory vs no-memory across sessions, and trigger-phrasing robustness. Protocols are immutable as of registration.
Authority expansion: index grown to 46 repositories across all stack layers (orchestration, evaluation, governance, memory, tools added as categories), 24 scored individual skills, 12 loop patterns, 10 memory patterns, 55 glossary terms, 15 templates, 18 guides. Every repo verified live; ragas and NeMo-Guardrails failed verification (moved) and were excluded; iannuttall/claude-agents indexed as Deprecated (archived upstream).
Index expanded to 22 repositories: added memory systems (mem0, letta, basic-memory), tool collections (MCP reference servers, serena, context7), workflow tooling (claude-task-master), instruction sets (awesome-copilot, Fabric, agent-rules), and two aggregators. All verified live; 12 new scorecards with evidence notes.
steipete/agent-rules indexed as the first Deprecated-tier entry: archived by its maintainer in May 2026. Kept with a warning per policy; deletion breaks citations.
Badge program launched: every fully-scored entry has an embeddable live SVG badge at /badge/[slug].svg. Comparison pages now generate for any scored pair. Index gained instant search. llms topic digests added under /llms/.
Methodology v1.0 published: 8 sub-scores, weighted composite, tier gates, null-renormalization for partial scores, adoption capped at 5%.
Initial index published: 10 repositories verified live against the GitHub API (metrics timestamped 2026-07-07), 8 fully scored, 1 partially scored, 6 individual skills scored, 4 loop patterns and 3 memory patterns documented.
Seed-list verification: Forward-Future/loop-library could not be verified to exist and was excluded. Loop patterns are sourced from verified frameworks instead.
Benchmark #1 (one-shot vs plan-execute-verify) protocol pre-registered. Results publish when runs complete; the protocol is immutable as of this entry.
Site launched: Field Guide (12 articles), glossary (29 terms), templates library (7), rankings (2), llms.txt suite, full schema markup, dark and light modes.