Evaluation Sets and Automated Checks

Why this matters

A team ships a ticket-summarisation feature after reviewing twenty outputs in a spreadsheet. They all looked fine. Two weeks later a prompt tweak meant to shorten summaries goes out, and the downstream routing service starts rejecting about one ticket in fifteen: the model has begun prefixing its JSON with "Sure! Here is the summary:". Nobody noticed, because nobody looked at the same twenty examples again, and even if they had, a human skimming for tone does not check whether a string parses.

This is the most common way language-model features degrade in production. Not a dramatic failure, but a quiet change in behaviour that nobody measured because the measurement only ever existed in someone's head. The fix is not a cleverer prompt. It is an evaluation set: a fixed, versioned collection of inputs with written expectations, and a harness that checks every output against them on every change.

Evaluation is now one of the most requested skills on AI engineering teams for exactly this reason. Models, prompts and providers change frequently. Without a repeatable test, every change is a guess.

The idea

An evaluation set is to a language-model feature what a unit-test suite is to ordinary code, with one important difference: the system under test is not deterministic, and many of its outputs have more than one acceptable form. That pushes you to separate two kinds of check.

Hard checks are properties that must hold for every output and can be verified by code: the output parses as JSON, a required field is present, a category belongs to an allowed set, a summary is under a length limit, no internal ticket ID or email address leaks into a customer-facing reply. A hard check either passes or fails. If it fails, the case fails, no matter how good the prose is.

Graded checks cover qualities that need judgement: is the summary faithful to the ticket, does it capture the customer's actual request, is the tone appropriate. These are scored against a written rubric, either by people or, with care, by another model. The next lesson covers model-based grading. Here, the point is that a rubric must be written down before you look at results, otherwise you will unconsciously grade to the outcome you wanted.

The set itself needs structure. Each case has an ID that never changes, an input, the expectations that apply to it, and a slice label such as billing, refunds or non-english. Slices matter because averages hide regressions. A change that improves the common case by three points and breaks a rare but expensive case by twenty can still raise the overall number.

Where do cases come from? Start with real traffic, anonymised and with consent where your policies require it. Add every production incident as a new case, so the same failure cannot ship twice. Then add deliberately difficult cases: very long inputs, ambiguous requests, mixed languages, inputs that contain instructions. A useful early target is fifty to two hundred cases with every important slice represented, rather than thousands of near-duplicates of the easy path.

Finally, the set is versioned alongside the prompt and code. A score only means something when you can say which cases, which prompt version, which model identifier and which sampling settings produced it.

Diagram

Worked example

You own a feature that turns inbound support tickets into a JSON object with two fields: category, one of billing, account, technical or other, and summary, at most 40 words, which a routing service and a human agent will both read.

You write the contract first. Hard checks: the output must parse as JSON; category must be in the allowed set; summary must be non-empty and within 40 words; and each case lists one or two keywords that any faithful summary must contain, such as "refund" for a duplicate-charge ticket. Graded check, for later: a five-point faithfulness rubric with anchored descriptions of what a 1, 3 and 5 look like.

You build 120 cases: 40 billing, 30 account, 30 technical, and 20 refunds, which you treat as their own slice because refund mistakes cost money. Each case gets a stable ID such as refund-017.

The baseline prompt passes 111 of 120 cases on hard checks, which is 92.5%. Then a colleague's "make it more concise" change produces 113 of 120. On the headline number, that is an improvement. On the per-case report, it is not: four cases that previously passed now fail, and six that previously failed now pass. Two of the four new failures are refund tickets where the summary no longer mentions the refund at all, and one is the "Sure! Here is the summary:" prefix, which breaks JSON parsing.

The decision is straightforward once the data is laid out this way. The change is not shipped as is. The prefix is fixed by tightening the output instruction and adding structured-output enforcement where the provider supports it. The refund cases are investigated, and a new rule is added to the prompt. The six newly passing cases are kept as evidence that the conciseness goal was reasonable. The rerun is compared case by case against the baseline, not just on the total.

Notice what made this possible: stable case IDs, checks written before the run, slices, and a report that shows which cases changed state. None of it needed a sophisticated tool.

Code

This standalone script runs the hard checks from the worked example on four recorded outputs. It uses only the Python standard library.

import json

ALLOWED = {"billing", "account", "technical", "other"}
cases = [
    {"id": "t1", "must_include": ["refund"],
     "output": '{"category": "billing", "summary": "Customer requests a refund for a duplicate charge."}'},
    {"id": "t2", "must_include": ["password"],
     "output": '{"category": "account", "summary": "User cannot reset password; the reset email never arrives."}'},
    {"id": "t3", "must_include": ["outage"],
     "output": '{"category": "technical", "summary": "Dashboard returns 500 errors since this morning."}'},
    {"id": "t4", "must_include": ["cancel"],
     "output": "Sure! Here is the summary: the customer wants to cancel."},
]

def check(case, max_words=40):
    try:
        data = json.loads(case["output"])
    except json.JSONDecodeError:
        return {"valid_json": False, "category": False, "keywords": False, "length": False}
    summary = str(data.get("summary", "")).lower()
    return {
        "valid_json": True,
        "category": data.get("category") in ALLOWED,
        "keywords": all(k in summary for k in case["must_include"]),
        "length": 0 < len(summary.split()) <= max_words,
    }

report = {case["id"]: check(case) for case in cases}
for name in ["valid_json", "category", "keywords", "length"]:
    passed = sum(result[name] for result in report.values())
    print(f"{name:11} {passed}/{len(cases)}")
print("failing:", [cid for cid, result in report.items() if not all(result.values())])

Expected output: valid_json 3/4, category 3/4, keywords 2/4, length 3/4, and failing: ['t3', 't4']. Case t3 parses correctly but fails the keyword check, which may mean either a weak summary or a keyword expectation that is too literal. Deciding which is part of the job. In a real harness, the outputs come from calling your model with a recorded model identifier and settings, and the report is saved with the prompt version.

Common mistakes

"We will build the evaluation set once the feature is stable." This is tempting because writing cases feels like overhead while the design is still changing. In practice the feature never feels stable, and the first serious regression arrives before the set exists. Start with twenty cases on day one and grow the set as you learn.

"A single overall score is enough to make a decision." This is tempting because one number is easy to put in a dashboard. Averages hide slice regressions and swaps, where some cases break while others improve. Always keep per-case and per-slice results, and review the cases that changed state.

"Exact-match comparison against a reference answer is the safest check." This seems rigorous, but language-model output varies in wording even when it is correct. Exact match produces failures that are not real, and teams learn to ignore the harness. Use exact match for structured fields and labels, and use property checks or rubrics for free text.

"Test cases should be the examples from the prompt." These are the inputs the model has effectively already seen. Scoring well on them says little about new inputs. Keep few-shot examples and evaluation cases separate.

Your turn

Pick one language-model feature you use or maintain. Write ten evaluation cases with stable IDs, at least two slice labels, and at least three hard checks that can be verified by code. Include one case taken from a real failure you have seen.

Extend the script above to load your cases from a JSON file and save the per-case report to another JSON file named after the prompt version.

Success criterion: you can change the prompt, rerun the script, and produce a list of exactly which case IDs changed from pass to fail and from fail to pass.

Recap

• An evaluation set is a versioned collection of inputs with written expectations, and every score refers to a specific set, prompt, model and setting.
• Hard checks are code-verifiable and fail the case on their own; graded checks need a rubric written before results are seen.
• Slice labels exist because averages hide regressions in rare but expensive cases.
• Every production incident should become a permanent evaluation case.
• A report that shows which cases changed state is more useful than a single headline number.

Sources

• Ribeiro et al. (2020), Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Motivates capability-specific test cases over single aggregate scores.
• Liang et al. (2022), Holistic Evaluation of Language Models. Describes evaluating language models across multiple scenarios and metrics rather than a single benchmark.
• OpenAI, Evals repository, accessed October 2026. An open-source example of a registry-based evaluation harness for language-model outputs.