Golden dataset

A golden dataset is a curated set of inputs with expected outputs or reference answers, used to test an AI application the same way every time. It turns subjective quality checks into repeatable regression tests and is the foundation of most LLM evaluation pipelines.

What goes into a golden dataset

• Real questions from users or support tickets, not only ones you invent.
• The expected answer, or the key facts a good answer must contain.
• For RAG apps, the source passage that supports the answer.
• Edge cases: ambiguous questions, out-of-scope questions and attempts to break the rules.

How to keep it useful

Start with a few dozen well-chosen cases, version the file alongside your code, and add every bug you find in production as a new case. Have a domain expert review the expected answers, because a wrong “golden” answer quietly teaches the wrong lesson.

Read the full guide: How to Test LLM Applications 2026 — Complete Guide for QA Engineers