Evals Are the New User Research
How to hold AI to a standard when it never repeats itself
Every AI team I have worked with hits the same wall around month three. The demo felt like magic. The launch went fine. Then quality starts slipping, and nobody can say how. Someone tweaks a prompt to fix one complaint, and quietly breaks nine other things nobody was watching. That is not the model's fault. The team simply never wrote down what good looks like.
Normal software is predictable. Give it the same input, you get the same output. So you test it: given this input, expect this exact answer. AI is different. It can answer the same question two ways, and both can be fine. So you cannot check for one exact answer. You check that the answer has the right qualities. That test is called an eval. Think of it like marking a hundred essays with a checklist, not marking a maths quiz against one answer key. And deciding which qualities count is a design call, not an engineering one.
Evals are the single most important lever for improving AI products. Teams that invest in them ship faster and with more confidence than teams that rely on vibes.
What an eval actually is
You need three things. A set of real inputs. A set of graders that score the answers. And a pass mark you refuse to ship below. That is the whole idea. The skill is picking inputs that look like real use, and writing graders that measure something a user would call quality.
How to build the set as a designer
The eval set is the whole game, and most people build it wrong by starting from imagination. Start from real failures instead. Go to your logs, your support tickets, and your own annoyed notes from using the thing, and pull out the answers that were genuinely bad. Those are your first cases. A set grown from real failures tests the ways your product actually breaks, not the ways you guessed it might.
Keep it small and mean. You do not need ten thousand cases to begin. You need fifty that hurt. A good eval case is adversarial, which means it aims straight at a weakness you have already seen: a question whose right answer is I do not know, an input in the wrong language, a request that should be refused. Fifty sharp cases catch more real regressions than a thousand easy ones, and a human can actually sit and read fifty. When a new bug slips through in production, add it to the set the same day, so the product can never regress into that exact mistake again without a test going red.
This is why a golden set beats a vibe check. A vibe check is opening the app, typing a few things, and deciding it feels fine. It is comforting and close to worthless, because you unconsciously type the inputs you know it handles. A golden set is a fixed, labelled list of inputs with the answer or the qualities you expect, written down once and run on every change. The vibe check tells you how you feel today. The golden set tells you whether you broke something since yesterday.
- Open the app and type a few things
- Inputs you already know it handles
- A verdict that shifts with your mood
- No record, so no way to compare builds
- A fixed list of labelled inputs
- Chosen from real, painful failures
- The same cases run on every change
- A number you can hold against yesterday
The three kinds of grader, and who owns them
- 01Fixed checks. Did it return valid data, stay fast enough, keep under the size limit, avoid banned phrases? Cheap, fast, run on every change. Engineers own these.
- 02AI-graded. Does the answer stick to the source it was given? Does the tone match the product? Did it say no when it should have? Here one AI scores the answer against a rubric. A rubric is just the marking scheme: the list of what earns a high or low score. Design should own the rubric, because the rubric is your product's taste written down.
- 03Human. A small sample checked by someone who knows the subject. Slow, costly, and impossible to replace. This is user research wearing a lab coat.
| Axis | Usability test | Eval |
|---|---|---|
| Sample size | 5 to 8 participants | Hundreds to thousands of cases |
| Cadence | Once a milestone | Every commit, automatically |
| Who runs it | A researcher, live | The build, then a human on the sample |
| What it catches | Confusion, intent, delight | Slipping quality, wrong refusals |
| Reusable? | Notes age quickly | The test set grows over time |
- The answer should be helpful.
- The tone should feel on brand.
- Refuse when appropriate.
- Avoid making things up where possible.
- Every fact comes from a source it was given.
- Second person, no exclamation marks, no waffle.
- Refuse any request for legal or medical advice.
- Score 1 if a claim is unbacked and a user could act on it.
Machine scoring versus human judgement
The middle grader, one AI scoring another, is the one people distrust most, and the research is kinder to it than you would guess. In 2023 a team tested using a strong model as the judge and measured how often it agreed with human raters. GPT-4 as a judge reached over eighty percent agreement with people, which is about how often two humans agree with each other. So an AI grader is not a toy. On a clear rubric it is roughly as consistent as a second human reviewer, and it never gets tired on case forty.
But it comes with biases you have to design around. The same study found the judge model tends to prefer the first answer it sees, prefer longer answers, and prefer answers from a model like itself. Leave those uncontrolled and your grader quietly rewards length and confidence over being right. This is why humans never fully leave the loop. Automated scoring runs on every build for pennies. Human judgement writes the rubric, checks a sample, and catches the failures the machine is biased to wave through.
Read your failures, do not chase one score
The trap with evals is polishing one average until it stops meaning anything. A single score of 0.87 tells you nothing you can act on. What actually moves a product is reading the failures. Take a hundred bad answers. Read every one. Group them by cause. The list of causes is usually shorter than people expect, and usually not what the team guessed.
That chart is the most useful thing you can make in AI work, and it is really a research finding. It tells you a fancier model would fix only twelve percent of your problem. Fixing how you fetch and quote sources, and when the AI says no, would fix over half. This is the same prioritising designers have always done with usability findings. Same craft, new ground.
Where designers plug in
- Curate the eval set. Which inputs stand for real users? The mix of inputs you test is the mix of users you are choosing to serve. That is a strategy call dressed up as a data task.
- Write the rubrics. Say what true to source means, what a fair no looks like, and how the product should sound when it is unsure. Make it clear enough that two people score it the same.
- Own the list of failure causes. Sit in on the review. Group failures by how much they hurt the user, not by tech cause, and the roadmap writes itself.
- Set the pass mark. Deciding which quality bar blocks a release is a call about risk and trust. Do not let it default to whatever number the build happened to hit.
export const faithfulness = {
name: "faithfulness",
scale: [1, 5],
rubric: `
5 - Every claim comes from the source. Nothing added.
4 - All claims backed; small rewording that changes no meaning.
3 - One unbacked but harmless line (e.g. a linking sentence).
2 - An unbacked claim a user could act on.
1 - Contradicts the source.
`,
blockReleaseBelow: 4.2,
}The honest risk: a green dashboard that lies
Here is the strongest case against evals, and it is real. An eval set rots. The day you write it, it mirrors your product and your users. Six months later the product has changed, people ask new things, and the set is quietly testing a world that no longer exists. A green scoreboard on a stale set is worse than no scoreboard, because it sells false confidence. You feel measured while flying blind.
There is a second trap. Teams start tuning the product to beat the set. Once a fixed list of cases becomes the target, people optimise for those exact cases, and the score climbs while real quality stands still. A measure that becomes a target stops being a good measure. The set goes green and users keep complaining, which is the most disorienting failure mode in this whole practice.
Neither problem kills evals. Both have the same answer: keep the set alive. Every week, fold in the newest real failures and retire cases that no longer reflect anyone. Hold back a slice of cases the product has never been tuned against, so you always keep one clean test. An eval set is not a monument you carve once. It is a garden you weed. Treat it as finished and it will lie to your face with a straight number.
Why this matters for your career
For twenty years the strongest knock on design was that you could not measure it. Evals hand us the opposite problem. An AI product is measured all the time. If designers are not in the room when those measures are set, the product gets tuned toward whatever engineers found easy to score. Usually that is helpfulness. And helpfulness on its own gives you a yes-man system that agrees with everything and is trusted by no one.