openai/evals
OpenAI's original eval framework and benchmark registry. Historically foundational; maintenance has visibly slowed as evaluation moved into newer tooling. Agentiquette scores it 68/100 (B-tier), methodology v1.0, scored 2026-07-07.
Key facts
| Category | Evaluation / observability |
|---|---|
| License | NOASSERTION |
| Maintainer | OpenAI |
| Compatible agents | generic |
| File patterns | |
| Triggering | manual-reference |
| GitHub metrics | 18,864 stars · 3,009 forks · 215 open issues (fetched 2026-07-07) |
| Last commit | 2026-04-14 |
| Created | 2023-01-23 |
Editorial analysis
Scored for what it is in 2026: the framework that established eval-as-YAML conventions, now in maintenance twilight while its ideas live on in promptfoo, deepeval, and vendor tooling. The registry remains a genuinely useful corpus of task definitions. New projects should read it and build elsewhere.
Risks
12-week commit gap at fetch time and an ecosystem that has largely moved on; treat as a reference corpus, not an active dependency.
Evidence notes: Maintenance: 12-week gap, community PRs unmerged. Eval readiness: the registry format remains exemplary. Adoption: historical stars, log-capped.
Install and first run
pip install evalsAlternatives in this category
Test harness for prompts, agents, and RAG pipelines: declarative test cases, assertions, matrix comparisons, and red-team scanning, runnable…
A pytest-style LLM evaluation framework: metric classes for correctness, hallucination, RAG quality, and agent task completion, written as…
Open-source LLM engineering platform: tracing, evals, prompt management, and metrics for agent systems. The observability layer most stacks…
Embed this badge
Maintainers of scored entries can embed their live badge; it updates when the score does and links back to this page.
[](https://agentiquette.com/index/repos/openai-evals)Frequently asked questions
Is openai/evals good?
openai/evals scores 68/100 (B-tier) on Agentiquette's methodology v1.0, scored 2026-07-07. Strengths and weaknesses are in the editorial analysis; the score is explained dimension by dimension in the scorecard.
What is openai/evals for?
OpenAI's original eval framework and benchmark registry. Historically foundational; maintenance has visibly slowed as evaluation moved into newer tooling.
Is openai/evals maintained?
Last commit 2026-04-14, 215 open issues at fetch time (2026-07-07). Maintenance is scored at 45 of 100.
Maintainer of openai/evals? Claim this listing and suggest corrections.