LLM-as-a-judge

LLM-as-a-judge is an evaluation method where one language model scores the output of another against a rubric, such as correctness or tone. It scales better than human review but can be biased or inconsistent, so judges should be calibrated against human-labelled examples.

How it works

You give a second model the question, the answer and a scoring rubric, and it returns a score, usually with a reason. Frameworks such as DeepEval (G-Eval) and Promptfoo (llm-rubric) build on this idea.

Pros and cons

• Pros: scores open-ended answers that exact-match checks cannot, and scales far better than human review.
• Cons: costs money per evaluation, can be biased (for example towards longer answers or its own model family), and is itself not fully consistent between runs.

Tips

• Write a clear rubric with examples of good and bad answers.
• Check the judge against a sample of human ratings before you trust it.
• Use a strong model as the judge and keep its settings fixed.

Read the full guide: LLM Evaluation Metrics: What They Measure and Which to Use (2026)