Every team building with language models eventually hits the same wall. A prompt change fixes one complaint and silently breaks something else. A new model version arrives with promises of better performance, and nobody can say whether it’s better for your use case. A stakeholder asks “how accurate is it?” and the honest answer is “it seems good”.
Evaluation — systematic, repeatable measurement of quality — is what gets you past that wall. This guide explains how to build evaluations that are useful, without turning it into a research project.
Why LLM evaluation is different
Traditional software tests check that a function returns an exact value. LLM outputs vary: the same correct answer can be phrased many ways, and quality has several dimensions — correctness, completeness, tone, format, safety. That doesn’t make evaluation impossible; it means you need a mix of techniques.
Step 1: Define quality for your feature
Start with the question: what does a good output look like, and what are the failure modes that matter? Involve the people who understand the domain — support leads, lawyers, analysts — not just engineers.
For a support assistant, quality might mean: correct according to policy, grounded in the help centre, appropriately concise, polite, and escalates when it should. Important failure modes might be: inventing a refund policy, giving account-specific information without verification, or failing to escalate an angry customer.
Write these down as explicit criteria. They become your rubric.
Step 2: Build a test set from reality
The test set is the foundation. Good sources include:
- Real user inputs from logs, tickets, search queries and chat transcripts.
- Known failures — every bug report becomes a test case.
- Edge cases identified with domain experts.
- Adversarial cases: attempts at prompt injection, off-topic requests, manipulation.
For each case, record what a good response must contain or avoid. That might be a reference answer, a list of required facts, a correct label, or a rubric-based expectation.
Start with 50–100 cases if that’s what you have; grow to several hundred over time. Coverage matters more than size — make sure each important category of input is represented.
Step 3: Choose evaluation methods
Use the cheapest reliable method for each criterion.
Deterministic checks. Fast, free and unambiguous. Is the output valid JSON matching the schema? Does it contain a required disclaimer? Is the chosen category correct? Is the answer under 150 words? Does it avoid forbidden phrases? Use these wherever possible.
Reference comparisons. For tasks with a known correct answer — extraction, classification, short factual answers — compare against references with exact match, field-level accuracy or fuzzy matching.
Model-graded evaluation. For subjective or open-ended criteria — helpfulness, faithfulness to sources, tone — use a capable model as a judge, with a specific rubric. “Rate helpfulness 1–10” is unreliable; “Does the answer state the refund window from the policy? Answer yes or no, citing the sentence” is much better. Binary or small-scale questions with clear definitions work best.
Human review. Essential for calibration and for high-stakes criteria. Review a sample regularly, especially after significant changes.
Step 4: Calibrate your judges
A model-graded evaluator is itself a model that can be wrong or biased — for example, favouring longer answers or answers similar to its own style. Before trusting it:
- Have humans grade a sample of outputs using the same rubric.
- Run the judge on the same sample.
- Measure agreement. If it’s low, refine the rubric or the judge prompt, or use a different judge model.
Re-check calibration occasionally, and whenever you change the judge.
Step 5: Evaluate RAG in two parts
For retrieval-augmented systems, separate:
- Retrieval quality. For each question, was the passage containing the answer retrieved? Metrics like recall@k tell you whether the model even had a chance.
- Generation quality. Given the retrieved context, was the answer correct, complete and faithful — meaning every claim is supported by the context?
This separation tells you where to fix things. Poor retrieval needs better parsing, chunking or search. Poor generation with good retrieval needs prompt or model changes.
Step 6: Evaluate agents on outcomes and trajectories
For agents, evaluate both the outcome (did the task get done correctly?) and the trajectory (did it use appropriate tools, in a sensible order, without unnecessary steps or unsafe actions?). Simulated environments with mock tools make agent evaluation repeatable. Track cost and step count per task as well; an agent that succeeds after forty tool calls has a problem.
Step 7: Run evaluations automatically
Evaluation is most valuable when it runs on every change:
- Store prompts, configurations and test sets in version control.
- Run the evaluation suite in CI when prompts, models, retrieval settings or code change.
- Set thresholds: a change that drops an important metric below a bar fails the build.
- Report a comparison against the current production version, broken down by category.
Tools such as Promptfoo, DeepEval, Ragas, Inspect and observability platforms like Langfuse or Phoenix make this practical. The tool matters less than the habit.
Step 8: Evaluate in production too
Offline tests can’t anticipate everything. In production:
- Sample live traffic and run automated evaluators on it.
- Collect user feedback — thumbs, edits, escalations — as quality signals.
- Monitor drift: new topics, new input types, changing distributions.
- Feed failures back into the test set.
The test set should grow continuously from real failures. Over time, it becomes one of your most valuable assets — it encodes what quality means for your product.
Common mistakes
- Evaluating on invented examples that are cleaner and easier than real inputs.
- One aggregate score that hides failures in important categories.
- Uncalibrated LLM judges treated as ground truth.
- Evaluating once at launch and never again.
- Optimising the metric instead of the outcome — for instance, rewarding length because judges prefer it.
A minimal starting kit
If all of this feels like a lot, start small. In a single afternoon a team can usually put together:
- Fifty real inputs, copied from logs or tickets, covering the main categories of use.
- A one-line expectation for each: the correct label, the fact the answer must contain, or the thing it must not do.
- A script that runs the current prompt and model over all fifty and applies simple checks.
- A spreadsheet or dashboard showing pass rates by category, saved for every run.
That’s enough to catch the worst regressions and to make the next model upgrade a measured decision. From there, grow the set from every bug report, add a calibrated model-graded check for the subjective criteria that matter most, and move the script into CI. Evaluation maturity is built incrementally; the important thing is to start before launch rather than after the first complaint.
Reporting results to stakeholders
Evaluation results are only useful if people outside the engineering team can act on them. Report in the language of the business: “the assistant answers 88% of billing questions correctly, up from 81%; refund-policy answers are still weak at 70%” is far more useful than an aggregate score. Show a handful of real examples of failures alongside the numbers, because they make the gap tangible and often prompt domain experts to clarify what the right answer should be. Keep a simple changelog linking each release to its evaluation results; over months it becomes the story of how the product improved.
What good looks like
A team with healthy evaluation practices can answer, within minutes of any change: did quality go up or down, in which categories, and at what cost? That confidence lets them upgrade models, refine prompts and ship features faster — and it gives stakeholders a real answer to “how good is it?”
Our LLM evaluation and guardrails service sets up exactly this foundation.