Evals, not vibes: measuring an LLM feature before you ship it

Most AI features ship because someone ran a handful of prompts, liked what came back, and called it done. That approach works right up until real users arrive with inputs nobody imagined, and then it fails in a way that's hard to diagnose because there was never a baseline to compare against.

An eval set is the fix. It is not complicated, it is not research infrastructure, and most teams building AI features still don't have one.

Why normal testing doesn't transfer

Conventional tests assert equality. Given this input, expect exactly this output. That model breaks for language features on three counts:

• The output is non-deterministic — the same input produces different text on different runs.
• Many different outputs are equally correct. There is no single right summary of a document.
• Quality is graded, not binary. An answer can be correct but too long, accurate but rude, helpful but subtly outside policy.

So the question changes from "is this output correct?" to "across a representative set of inputs, how often is the output acceptable, and is that rate getting better or worse?"

What an eval set actually is

A collection of inputs, each with a definition of what an acceptable output looks like, plus a way of scoring. That's all.

Fifty to two hundred cases is enough to be useful for most product features. The value is not in volume — it's in the cases being representative of what users actually send, including the ugly ones.

Four ways to score, in order of preference

1. Deterministic checks

Where you can assert something objectively, do. Is the output valid JSON? Does it contain a required field? Is it under the length limit? Does it cite a document that actually exists in the corpus? Does it avoid a forbidden term?

These are cheap, fast, deterministic and catch a surprising share of real failures. Always start here and push as much as possible into this category.

2. Reference comparison

For tasks with a defensible correct answer — extraction, classification, structured retrieval — write the expected answer once and compare. Exact match where the output is structured, fuzzy match where it's text.

3. Model-graded scoring

Use a model to grade against a rubric. This works, with two conditions that are non-negotiable: the rubric must be specific enough that two humans would agree on it, and you must periodically check the grader against human judgement.

"Rate the helpfulness from one to ten" is not a rubric. "Does the answer address the question asked, cite at least one source from the provided documents, and avoid recommending an action outside the supported list? Yes or no for each" is.

4. Human review

Expensive, slow, and irreplaceable as a calibration layer. Not for every run — for a sample, periodically, to confirm the automated scores still correlate with what a person would say.

The cases people forget

Eval sets built from happy paths give false confidence. These are the categories most often missing:

1. 1Inputs with no good answer. What should the feature do when the information genuinely isn't available? "I don't know" is usually the correct output and models are reluctant to produce it.
2. 2Adversarial inputs. Prompt injection attempts, instructions embedded in retrieved documents, users trying to make the feature say something embarrassing.
3. 3Out-of-scope requests. Legal, medical or financial questions arriving at a product not built to answer them.
4. 4Boundary sizes. An empty document. A document at the context limit. A single-word query.
5. 5Language and encoding. Non-English input, mixed scripts, emoji, right-to-left text.
6. 6Ambiguous inputs where the correct behaviour is to ask a clarifying question rather than guess.

Running it

Wire the eval set into CI and record the score on every change to a prompt, a model version, a retrieval configuration or a chunking strategy. Then treat the number the way you'd treat a test suite: a drop is a blocking failure requiring explanation, not a metric to note and move past.

The value compounds over time. The first run tells you very little in absolute terms. The fiftieth tells you exactly which change broke things, which is the entire point.

You are not trying to prove the feature is good. You are trying to notice, quickly, when it stops being good.

Cost and latency belong in the eval too

Quality is not the only dimension that regresses. A prompt change that improves accuracy by two points while tripling token spend is usually a bad trade, and you will not notice it for a month if you're only tracking quality.

Record tokens in, tokens out, and wall-clock latency alongside the score. Set a budget for each and fail the build when it's breached, the same as for quality.

The honest reason most teams skip this

Building an eval set takes a couple of days and produces nothing a stakeholder can see in a demo. Skipping it takes zero days and the demo looks identical.

The difference only shows up later — at the point where somebody asks whether last week's prompt change made things better or worse, and the honest answer is that nobody can tell.