LLM & AI Testing
How to test LLM applications, chatbots and RAG pipelines: evaluation metrics, hallucination checks, golden datasets and CI quality gates, with working examples in Promptfoo, DeepEval and RAGAS.
Start here
New to this topic? Read these in order.
- 1How to test LLM appsThe overview: what to test, how to build a golden dataset and where each tool fits.
- 2Evaluation metricsWhat the common metrics measure and which ones to start with.
- 3Promptfoo tutorialBuild your first LLM test suite step by step.
- 4What is RAGASHow to test retrieval-augmented (RAG) apps.
- 5Promptfoo vs DeepEvalWhich framework to pick for your team.
What LLM testing covers
LLM testing checks whether an AI application gives answers that are correct, grounded and safe. The same prompt can return different text each time, so instead of exact-match assertions you score answers against expectations. In practice that means four steps: build a small golden dataset of real questions and expected answers; pick metrics such as faithfulness, answer relevance and hallucination checks; run them automatically with a framework like Promptfoo, DeepEval or RAGAS; and add a quality gate in CI that fails the build when scores drop.
These guides are written for QA engineers and SDETs moving into AI testing. Each one says what we ran ourselves and what we only researched.
