The Eval Is the Spec: Deciding What "Working" Means Before You Ship AI

The first thing that breaks when a team ships its first real AI feature isn't the model. It's the definition of "done." For a normal feature, done is legible: the button saves the form, the list paginates, the test is green. For a feature where a language model sits in the middle, "working" isn't a state you can point at. It's a distribution. The same input can produce a good answer today and a subtly worse one after a prompt tweak, a model version bump, or a Tuesday. If your definition of done is "it looked right in the demo," you don't have a spec. You have a screenshot.

So on my team the rule is simple and slightly annoying: you don't get to build the AI feature until you've written the eval that decides whether it works. Not after. First. The eval is not a testing chore you bolt on before release — it is the specification, written in the only language that survives contact with a probabilistic system: examples with expected behavior, and a way to score how far you are from it.

"Working" is a distribution, not a demo

The demo is the most dangerous artifact in AI development because it is a sample of size one, chosen by the person most motivated for it to succeed. Every feature demos well; that's what a demo is for. The question that actually matters — how does this behave across the messy, adversarial, half-typed, wrong-language, edge-of-the-schema inputs real users bring — is invisible in a demo and completely visible in an eval set. Writing the eval first forces the argument that the demo lets you skip: what, concretely, are the cases this has to get right, and what does "right" mean for each of them?

A demo proves the feature can work once. An eval decides whether it works. Those are not the same claim, and only one of them ships.

Writing the eval first surfaces the real disagreements

The most useful side effect of writing the eval before the code is that it drags the vague parts of the feature into the open while they're still cheap to fix. You sit down to write twenty expected outputs and discover that two engineers, the PM, and the designer each quietly held a different definition of a correct answer. That disagreement was always there. Building first just hides it until it's expensive — until it's a shipped behavior someone has to defend to a user. The eval is where you spend an afternoon arguing about what "good" means, on purpose, before any of it is load-bearing.

It also changes the shape of the eval set. When the eval is an afterthought, it fills up with happy-path cases that were always going to pass — a green bar that certifies nothing. When it's the spec, you write the cases you're actually afraid of first: the ambiguous input, the prompt injection, the empty state, the input in a language you didn't plan for. The eval set becomes a written record of everything the team decided the feature must not do — which is exactly the thing a demo can never show you.

Scores over verdicts

A conventional test suite gives a verdict: pass or fail. An eval gives a score, and the difference matters. A probabilistic feature is never going to be 100%, and pretending otherwise leads to the worst failure mode — a team that quietly stops measuring because the number is never green. So we don't ask "does it pass?" We ask "what's the pass rate on the cases we care about, and is that rate going up or down since the last change?" The threshold is a product decision, made in the open: this feature ships at 94% on the core set with a deterministic fallback for the rest, and we know exactly which 6% we're choosing to catch a different way.

That number becomes the thing you defend a change against. Someone rewrites a prompt to fix one annoying case; the eval tells you within minutes whether they fixed one case and broke four others. Without it, prompt engineering is just vibes and confidence — you change the wording, the one input you happened to retry looks better, and you ship a regression you won't hear about until a user does. The eval is the only mechanism I've found that makes iterating on a non-deterministic system feel like engineering instead of gambling.

What this looks like on a 12-engineer team

Concretely: an AI feature doesn't get an implementation ticket until it has an eval ticket, and the eval ticket owns the definition of "working" — the case set, the scoring, and the threshold we'll ship at. The person building the feature usually isn't the person who wrote the first draft of the eval, because it's healthier when the spec isn't authored by the same optimism that will build against it. When the model version changes underneath us — and it will, without asking — the eval is what tells us whether our feature got better, got worse, or moved in ways we now have to have an opinion about. The alternative is finding out from a support ticket, which is the most expensive place to learn what your spec was.

None of this slows us down the way it sounds like it should. Writing the eval first feels like overhead right up until the first time a "small prompt tweak" silently degrades a feature and the eval catches it before the release does. After that, nobody on the team wants to build an AI feature without one. The eval isn't the thing that proves you're finished. It's the thing that let you know what finished meant in the first place.

If you're standing up eval practice on a team that's shipping real AI features, I'm happy to compare notes. Reach me at arun@arunjalota.com.