You changed the prompt. How do you know you didn't break it?
By Lindsey J
A product team ships a new system prompt for its AI agent on a Tuesday afternoon. The demo looks clean. Nothing in the release checklist catches what changed for the other nine hundred conversations the agent already handles that day.
That gap is not a testing problem in the usual sense. It is a design problem. Nobody defined what still works means before the change went out.
What an eval actually is
An eval is a small, repeatable test suite built from real cases. It is not a demo script and it is not a vibe check run by whoever shipped the change. Anthropic's own engineering team puts it plainly: "Evals make problems and behavioral changes visible before they affect users, and their value compounds over the lifecycle of an agent."
The distinction matters because an agent is not a single turn chatbot. It calls tools, holds state across a conversation, and can take an action a person then has to live with. Testing it the way you would test a static model misses exactly the failures that matter: the ones that only show up three tool calls in.
Start smaller than feels responsible
Anthropic recommends starting an eval set with 20 to 50 simple tasks pulled directly from real failures, not a theoretical suite built to cover every case up front. OpenAI frames the same instinct with its own structure: specify what success looks like using a small set of examples it calls a golden set, measure the system against real conditions, then improve from there, starting with something like 50 to 100 real outputs rather than waiting for a complete framework.
The exact numbers differ. One is describing an agent task suite. The other is describing a general output set. The instinct underneath is the same: pull the first cases from what the system has actually done, not from what you imagine it might do.
The step most teams skip
Anthropic's own team found that Opus 4.5 initially scored 42 percent on the CORE-Bench benchmark. An Anthropic researcher traced the low score to rigid grading, ambiguous task instructions, and stochastic tasks that could not be reproduced exactly. Fixing those issues and giving the model a less constrained scaffold brought the same model's score on that same benchmark up to 95 percent. Read that carefully: the model had not changed. What changed was how it was tested.
A dashboard number is only as good as whatever produced it. Read a sample of real transcripts by hand every time the eval set or the grading logic changes. Don't wait until a number looks wrong to check for a bad grader.
The metric this lands in
Frame an eval suite as insurance against a number a product team already tracks: the rollback rate on feature releases, or the count of incidents opened against a specific agent in the first week after a change ships. An eval set built from real failures catches the regression that would otherwise show up in that number first, after a user has already hit it.
Before the next change to an agentic feature ships, run this self-check:
Self-check before shipping an agent feature change
- Do you have at least 20 real failure cases pulled from production, not invented ones?
- Does your test set include cases where the agent should not act, not only where it should?
- Has a person read a sample of the actual transcripts in the last month?
- Would this catch a silent drop in accuracy before a user does, or only after?
The decision rule
If a change to a prompt, a tool definition, or a model version affects a feature that takes an action rather than only replying, and the eval set for that feature has fewer than the 20 to 50 real failure cases Anthropic recommends as a starting point, build the set before the change ships. Not after.
A worked example
Picture a support team that ships an AI agent to triage incoming tickets and route the urgent ones to an on call engineer. A routine model upgrade goes out on a Friday. The upgrade is faster and cheaper. Every existing dashboard, including ticket volume and response time, looks the same or better.
What the dashboards do not show is that the new model has quietly stopped flagging a specific category of emergency ticket, the ones written in short, panicked fragments rather than full sentences. Nobody notices for a week. The team's metrics track volume and speed. They do not track whether the right ten tickets out of ten thousand got flagged. A 20 task eval set built from last quarter's real misroutes, including a few of those short, fragmented emergency tickets, would have caught the regression before the upgrade shipped, not after.
FAQ
What is an AI eval, in plain terms?
A structured, repeatable set of test cases, pulled from real system behavior, checked against a clear standard for a correct response. It runs the same way every time, so a team can tell whether a change made things better or worse.
How many test cases does a team actually need to start?
Anthropic suggests 20 to 50 tasks drawn from real failures for an agent eval. OpenAI suggests 50 to 100 real outputs as a starting point for a general eval. Both are small enough to build in a week, and both are meant to grow from there.
Can user feedback replace a formal eval set?
No. User feedback only surfaces the failures a user noticed and reported. An eval set is built to catch the failures nobody has noticed yet, including the ones that never generate a support ticket at all.
Sources
- Anthropic, Demystifying evals for AI agents, January 2026
- OpenAI, How evals drive the next chapter in AI for businesses, November 2025
For more on what happens after an AI feature ships, see Your AI will be confidently wrong. Design for it and Your AI pilot works. Why won't it scale?.
A team that cannot answer the self-check above for its last AI feature change is not ready to ship the next one with confidence. Talk to Aero about building the eval suite before the next release, not after the incident.
