Your AI agent works great in the demo. It hasn't met your real task yet
By Lindsey J
A demo is usually one step. A real workflow rarely is. That gap explains why an AI agent that looks flawless during a walkthrough starts missing once it runs inside actual production.
Research published in March 2025 by METR, the nonprofit that measures what frontier AI models can do, gives this gap a name: time horizon. The metric is the length of task, measured in the time it would take a human professional, that a model can complete at a given success rate.
Why AI agent reliability drops as tasks get longer
METR's benchmark found that the frontier models it tested succeed on almost 100 percent of tasks that take a human professional less than 4 minutes. On tasks that take a human more than about 4 hours, the same models succeed less than 10 percent of the time. Models have improved since that study. The shape of the curve is the part that matters here.
The same research found that the task length a model can complete at 50 percent reliability doubled roughly every 7 months over the six years before the study. That trend explains why an agent that could not be trusted with a multi-step task last year might handle it acceptably now. It also means the wall moves out over time. It does not go away.
The mistake is measuring reliability once
Most teams test an agent's accuracy against the length of task in the demo, get a strong number, and stop measuring. The number stays true for that one workflow only. Every time a team chains on one more step, the task the agent is actually doing gets longer, and the reliability measured against the shorter version stops applying.
If a team has not measured an agent's success rate at the exact step count its production task takes today, it does not know that agent's reliability. It knows the agent's reliability on a different, shorter task.
Picture a support agent that drafts a single reply to a single ticket. Picture the same agent chained into a flow that reads the ticket, checks the account, issues a refund, and updates the subscription: four steps instead of one. The per-step accuracy can be identical in both cases. The end-to-end success rate is not, because each step's error compounds into the next one.
A decision rule for where to add a checkpoint
- Measure the agent's success rate at the exact step count the production task takes today.
- Find the step where the success rate crosses below what the product can tolerate. That threshold depends on how reversible the output is.
- Put the checkpoint at that step, before the agent commits to something the user cannot easily undo.
The metric a checkpoint protects is the downstream number a team already tracks: refund reversals, support escalations that need a person to unwind an agent's action, or the churn that follows a customer discovering an agent acted without asking first.
Why this is a design call, not only an engineering one
Where the checkpoint lands changes the product experience. It decides what the user sees before a refund goes out, what they can still edit, and what undo means once the agent has already acted. Agentic products live or die at the approval step covers what has to be true once that approval step exists and a user has to trust it.
A design system that can express a checkpoint state is what makes this decision rule usable across a whole product, one screen at a time.
Run this before the next agent release
- How many discrete steps does the agent take between a user's request and the output they see?
- What is the agent's measured success rate at that exact step count, measured this quarter?
- Where does a failure get caught: before the user sees it, or only after?
- Which step in the chain is the last one that could be undone without help from a person?
- Has the number of steps in this workflow grown since the last time its success rate was measured?
FAQ
How do I measure an agent's success rate at a given task length
Track outcomes against the actual number of steps used in production. Re-measure any time a step gets added to the chain.
Does adding a checkpoint slow the product down
Not if it only gates the step where an error becomes hard to undo. Most chains need one checkpoint, placed at the point where the task stops being easily reversible.
Is this only a concern for coding agents
Not only. METR measured software and reasoning tasks, so its numbers do not transfer directly to other work. Still, expect a similar shape anywhere an agent chains several steps together, such as support, sales operations, or back-office workflows. Measure it on your own workflow before you trust it.
An agent readiness review starts with this exact measurement: where a product's task chains sit on the reliability curve today, and where a checkpoint belongs before the next release. Talk to Aero about a review.
