Write the evaluation set while the AI feature is still moving

A lot of teams leave evaluation until the prompt and interface feel finished. I'd rather start collecting cases while the feature is still moving, because the useful ones turn up during development anyway: a developer changes the prompt after an awkward output, a subject-matter reviewer corrects an answer, or a test user has to explain what the feature misunderstood. If those examples stay in chat threads and scratch notes, they're gone, and the next change can quietly bring the same problem back.

The first record can be small. Input, relevant source material, observed output and a plain description of the problem, plus the reviewer's corrected result if there is one and any conditions that matter, like the user role, rule version or document date. "Answer was poor" won't help anyone in a month. "The answer used a superseded policy even though the current policy was present" is something you can test. Use approved, redacted or synthetic inputs where you can, and if a real case has to stay restricted, keep it in a controlled test environment instead of a handy shared spreadsheet.

Before tuning the prompt until that one output reads well, write down what success means in a way that applies beyond the example. Some checks can be exact: a classification from an allowed set, a calculated value that matches the versioned rules, a citation pointing to a source that was actually provided, an action within the user's permissions. Others need judgement, and a short scoring guide helps reviewers apply the same criteria, including what should be escalated instead of answered confidently. I'd also note when several answers are acceptable, since treating one preferred sentence as the only right answer rewards imitation.

The team will look at development cases while they change things, which is fine. Passing them tells you the known problems are fixed. A smaller held-back set that nobody tunes against tells you whether the fix carries over to similar work. It doesn't need to be elaborate at first, but you do need to record which cases shaped the change and which were used to check it afterwards, otherwise your pass rate mostly describes examples you've already rehearsed. Watch for near-duplicates too. Ten versions of the same easy request can make things look stable while one rare, consequential case never gets tested.

Each result should record the model, system instructions, retrieval settings, tool versions, rule versions and sampling controls, with the raw output kept next to the check result. Where a search index or connected service can shift between runs, I'd use fixtures or snapshots and run live integration checks separately. When I read results I want to see which task failed and how, because a formatting slip and a confident answer from the wrong source need very different fixes. Review effort counts as well. If accuracy goes up but every answer takes longer to check, the product trade-off has changed.

Retire invalid cases with a note instead of deleting them, and have the pull request or release check say which evaluation version ran, on which configuration, and which failures are still accepted. None of that needs the feature to feel finished first. The hard cases turn up while it's still moving, and that's when they're easiest to capture. So next time an example makes someone retune the prompt, turn it into a repeatable fixture before the discussion moves on, and you'll be able to tell whether the next change fixes the task or just tidies up the latest demo.