promptfoo/promptfoo
Test harness for prompts, agents, and RAG pipelines: declarative test cases, assertions, matrix comparisons, and red-team scanning, runnable in CI. Agentiquette scores it 83/100 (A-tier), methodology v1.0, scored 2026-07-07.
Key facts
| Category | Evaluation / observability |
|---|---|
| License | MIT |
| Maintainer | Promptfoo Inc |
| Compatible agents | generic |
| File patterns | |
| Triggering | manual-reference |
| GitHub metrics | 22,994 stars · 2,054 forks · 401 open issues (fetched 2026-07-07) |
| Last commit | 2026-07-07 |
| Created | 2023-04-28 |
Editorial analysis
The closest thing the ecosystem has to a standard test runner for the evaluation layer, and the tool this site's own benchmark protocols reference. Declarative YAML cases make evals diffable and reviewable, which is the property that separates evaluation from vibes. The red-team scanner doubles it as governance tooling.
Risks
LLM-graded assertions inherit grader bias; pin grader models and audit rubrics like code.
Evidence notes: Eval readiness: the product is eval readiness; assertions are checkable by construction. Docs: init-to-first-eval ran as written. Safety: red-team mode scans for injection and leakage classes.
Install and first run
npx promptfoo@latest initAlternatives in this category
A pytest-style LLM evaluation framework: metric classes for correctness, hallucination, RAG quality, and agent task completion, written as…
Open-source LLM engineering platform: tracing, evals, prompt management, and metrics for agent systems. The observability layer most stacks…
OpenAI's original eval framework and benchmark registry. Historically foundational; maintenance has visibly slowed as evaluation moved into…
Embed this badge
Maintainers of scored entries can embed their live badge; it updates when the score does and links back to this page.
[](https://agentiquette.com/index/repos/promptfoo)Frequently asked questions
Is promptfoo/promptfoo good?
promptfoo/promptfoo scores 83/100 (A-tier) on Agentiquette's methodology v1.0, scored 2026-07-07. Strengths and weaknesses are in the editorial analysis; the score is explained dimension by dimension in the scorecard.
What is promptfoo/promptfoo for?
Test harness for prompts, agents, and RAG pipelines: declarative test cases, assertions, matrix comparisons, and red-team scanning, runnable in CI.
Is promptfoo/promptfoo maintained?
Last commit 2026-07-07, 401 open issues at fetch time (2026-07-07). Maintenance is scored at 92 of 100.
Maintainer of promptfoo/promptfoo? Claim this listing and suggest corrections.