“It seems to work” isn’t a standard
Most LLM applications are launched after someone tried twenty questions and liked the answers. Then a prompt is tweaked, a model version changes, or new documents are added — and nobody knows whether things got better or worse until a customer complains. Evaluation turns that guesswork into measurement.
Building an evaluation that means something
A useful evaluation reflects real use. We build datasets from production logs, support tickets and expert input, covering common cases, known hard cases and adversarial inputs. For each case we define what a good response looks like: a reference answer, required facts, forbidden content or a scoring rubric. For RAG systems we measure retrieval separately from generation, because the fixes are different.
Evaluators are layered: cheap deterministic checks first (format, required fields, banned phrases), then model-graded rubrics calibrated against human ratings. The result is a dashboard that tells you, for each change, what improved, what regressed and by how much.
Red-teaming
Before launch we try to break the system the way a curious or malicious user would: prompt injection via uploaded documents, attempts to extract the system prompt or other users’ data, jailbreaks toward harmful content, and social-engineering the assistant into unauthorised actions. Findings are fixed and turned into permanent test cases.
Guardrails at runtime
Evaluation catches problems before release; guardrails catch them live. Depending on risk, we add input screening for injection and abuse, PII detection and redaction, topic restrictions, output checks against policy, and verification of factual claims against retrieved sources. Guardrails complement good architecture — they are not a substitute for limiting what the model can do.