The product team ran evaluations for three weeks. The agent scored 91% accuracy across a test set of 4,000 examples. Everyone felt good about it. They shipped.
Two weeks after launch, a pattern emerged in the support queue. The failures weren't random. They clustered: low-frequency request types, ambiguous phrasing at the edges of the training distribution, inputs that were technically valid but structurally unusual. Users hitting those inputs got wrong answers, sometimes confidently delivered wrong answers. The 91% headline had nothing to say about any of that.
This is the inverted U problem. Performance curves that look healthy in aggregate can hide a valley at the extremes. An agent that handles 91 out of 100 cases correctly can still fail systematically on the cases that matter most to a particular user or workflow. Aggregate metrics flatten the distribution. They tell you how the middle performs. They say almost nothing about the edges.
High average accuracy is not the same as reliable performance, and the difference shows up exactly where you can't afford it.
Where the accuracy number comes from
Most evaluation sets are built to resemble the expected distribution of inputs. That is a reasonable starting point. The problem is that "expected distribution" tends to mean "things we thought to include when we built the test set."
Edge cases, by definition, are underrepresented. A 4,000-example test set built from production logs will reflect whatever your early users happened to send. If those early users were relatively homogenous, if they used the product in expected ways, and if they self-selected out when they hit friction, then the test set is a flattering sample. It confirms that the agent works for the people most likely to succeed with it.
The users who generate support tickets are a different population. They ask questions in ways the training set didn't anticipate. They combine parameters that the evaluation set treated as independent. They interpret the product's scope differently than the team did. The 91% figure doesn't reach them.
This is not a failure of diligence. It is a structural property of how aggregate metrics work. Fixing it requires a different evaluation architecture, not a bigger test set.
Four layers that address different failure modes
The goal of a more robust architecture is not to eliminate all failure. That is not achievable. The goal is to make failures visible before they compound, and to prevent agents from acting confidently in domains where they are not reliable.
Progressive autonomy maps agent action scope to demonstrated reliability. An agent doesn't start with full execution rights. It starts narrow: read access, low-stakes decisions, single-step tasks with human review. As reliability is confirmed across a specific task class, scope expands for that class. This matters because aggregate accuracy hides which task classes the agent actually handles well. Progressive autonomy forces you to find out at a granular level before granting broader action.
The operational cost is real. You need a way to classify tasks, track reliability by class, and manage the handoff logic when an agent hits the edge of its granted scope. For small teams, that infrastructure is not free. But the alternative is discovering at scale that your agent was confident and wrong in a domain you assumed it could handle.
Deterministic validation is a layer of rules the agent cannot override. These are not soft guidelines or prompting constraints. They are hard checks: input validation against a schema, output ranges that fail if exceeded, action types that require a second confirmation step regardless of model confidence. The model makes a recommendation; the validator decides whether that recommendation is allowed to execute.
This layer catches a specific failure type: cases where the model is confidently wrong. High confidence is not the same as high accuracy, particularly at distribution edges. A model that returns a confidence score of 0.97 on an out-of-distribution input is not telling you it is right. It is telling you it produced an answer that looks a lot like its training examples. Deterministic validation doesn't care about the confidence score. It checks the output against rules that don't bend.
False-positive flywheels are feedback mechanisms built to surface over-confidence. When an agent acts on an input and the outcome is wrong, that failure needs to flow back into the evaluation system. Not as a manual review process, but as a structured signal: this input type, this output, this outcome. Over time, that signal builds a picture of where the agent's confidence is miscalibrated.
Catching individual failures is only part of what this layer does. Its real value is identifying patterns in where the agent over-extends. If failures cluster around a particular input structure, that is information you can act on: tighten validation rules for that class, reduce autonomy scope, or add a human review step. Without a flywheel, each failure is a one-time event. With one, failures become a data set.
Factorial stress testing is the evaluation practice of breaking the edges on purpose before you ship. Instead of testing inputs in isolation, you combine parameters that your normal test set treats independently. Edge-case inputs with unusual phrasing. Valid but low-frequency request types. Inputs that sit at the boundary of two task classes. High-volume edge cases run concurrently.
The point is not to find every possible failure. It is to find the failure modes that are systematic. Random failures are hard to prevent and usually low-impact. Systematic failures, where the agent fails reliably on a particular input class, are high-impact and fixable. Factorial testing surfaces the systematic ones before users do.
What each layer costs
Progressive autonomy adds operational complexity. You need task classification, reliability tracking by class, and scope management logic. For a solo developer or small team, that is real work before you get any production value from the agent. The tradeoff is deploying with confidence in specific task classes rather than hoping aggregate accuracy translates.
Deterministic validation requires you to know in advance what the rules are. That sounds obvious, but it forces a conversation many teams skip: what are the hard limits on what this agent is allowed to do, and what counts as an invalid output? If you haven't answered those questions, you can't build the validator. The process of building it is useful independent of the technical implementation.
False-positive flywheels require feedback infrastructure. The agent's actions need to be logged, the outcomes need to be tracked, and the signals need to flow back into the evaluation system in a structured way. That is not complex to build, but it is work that tends to get deprioritized when teams are focused on shipping features.
Factorial stress testing takes time that teams rarely budget for. The standard approach is to run the existing test set, see a high number, and ship. Running factorial combinations across parameter space takes longer and often surfaces failures that are uncomfortable to sit with close to a launch date. Teams that skip it usually do so because the test set gave them the answer they wanted.
Where to start
Build the deterministic validation layer first, before anything else. It is the one layer that doesn't require historical data, doesn't require feedback infrastructure, and doesn't require you to have already deployed. It only requires you to answer the question: what should this agent never be allowed to do, and what counts as a structurally invalid output?
That question surfaces your actual assumptions about the system. The answers become your first validation rules. Start there, then add the other layers as you accumulate production data and the feedback infrastructure to support them.
Aggregate accuracy will continue to look fine. The point of this architecture is to build a system that still performs when aggregate accuracy is the wrong metric to be looking at, which is most of the time the system is actually under stress.