DeepEval
Pytest-style framework for evaluating LLM outputs with built-in metrics — hallucination, answer relevancy, faithfulness, G-Eval, and more.
llm-evaluation-frameworksRecently released
72
Hero Score
Popularity
72
Performance
70
Ecosystem
75
Maturity
69
Dev Experience
75
⭐ 18,767 stars⬇ 786.1K downloads/wkFirst release: Aug 2023Last release: Sep 2026
Async Support: YesPlugin Extensions: HighSpeed: MediumDoc Quality: HighLearning Curve: Easy
Pros
- • Pytest-style API makes LLM evals feel like familiar unit tests
- • Rich built-in metric library (G-Eval, hallucination, faithfulness, relevancy)
- • CI-friendly with dataset support and Confident AI dashboard integration
Cons
- • Most metrics rely on LLM-as-judge — token cost and latency add up
- • Judge-model choice and prompts can shift scores between runs
- • Customizing metrics beyond built-ins requires reading the internals