confident-ai/deepeval
A pytest-style LLM evaluation framework: metric classes for correctness, hallucination, RAG quality, and agent task completion, written as unit tests. Agentiquette scores it 80/100 (A-tier), methodology v1.0, scored 2026-07-07.
Key facts
| Category | Evaluation / observability |
|---|---|
| License | Apache-2.0 |
| Maintainer | Confident AI |
| Compatible agents | generic |
| File patterns | |
| Triggering | manual-reference |
| GitHub metrics | 16,697 stars · 1,624 forks · 351 open issues (fetched 2026-07-07) |
| Last commit | 2026-07-06 |
| Created | 2023-08-10 |
Editorial analysis
The pytest framing is the insight: evaluation adoption fails on workflow friction, and putting LLM checks where engineers already run checks dissolves it. Agent-specific metrics (task completion, tool correctness) are newer and thinner than the RAG metrics, which are mature. Pairs naturally with promptfoo rather than competing.
Risks
Metric quality varies: the classic assertion metrics are solid, the more judgmental ones need calibration against your own labels before you trust a red/green.
Evidence notes: Eval readiness: metrics are assertable and CI-native. Docs: quickstart ran as written. Reliability: deterministic metrics strong; LLM-judge metrics documented with their variance caveats.
Install and first run
pip install deepevalAlternatives in this category
Test harness for prompts, agents, and RAG pipelines: declarative test cases, assertions, matrix comparisons, and red-team scanning, runnable…
Open-source LLM engineering platform: tracing, evals, prompt management, and metrics for agent systems. The observability layer most stacks…
OpenAI's original eval framework and benchmark registry. Historically foundational; maintenance has visibly slowed as evaluation moved into…
Embed this badge
Maintainers of scored entries can embed their live badge; it updates when the score does and links back to this page.
[](https://agentiquette.com/index/repos/deepeval)Frequently asked questions
Is confident-ai/deepeval good?
confident-ai/deepeval scores 80/100 (A-tier) on Agentiquette's methodology v1.0, scored 2026-07-07. Strengths and weaknesses are in the editorial analysis; the score is explained dimension by dimension in the scorecard.
What is confident-ai/deepeval for?
A pytest-style LLM evaluation framework: metric classes for correctness, hallucination, RAG quality, and agent task completion, written as unit tests.
Is confident-ai/deepeval maintained?
Last commit 2026-07-06, 351 open issues at fetch time (2026-07-07). Maintenance is scored at 90 of 100.
Maintainer of confident-ai/deepeval? Claim this listing and suggest corrections.