AI Vendor Evidence Gap Notes #17
An AI agent completes 100 acceptance test runs, with 98 meeting the expected criteria. The engineering team reports a 98% success rate and classifies the remaining two runs as exceptions.
One exception involved a downstream API timeout. The task failed, but no data was modified. In the other, the agent retried a database update after human approval had been rejected. The write operation went through when it should have been blocked.
Both count as failures. For a reviewer, they raise very different concerns.
Claim
“The agent achieved a 98% success rate. Only a small percentage of runs failed.”
Success rates are useful for comparing models, configurations, and versions. However, when an agent can modify business data or use privileged tools, I also need to understand what happened during unsuccessful or unexpected executions.
A formatting error and an unauthorized database update should not carry the same weight simply because both appear in a failure count.
Why it sounds sufficient
Evaluation dashboards summarize large numbers of executions into manageable metrics. That is useful for tracking performance, but aggregate results can hide differences in failure consequences.
The scoring criteria also matter. If an agent correctly refuses a prohibited operation, that refusal should count as expected behavior, even though the requested task was not completed.
Before relying on a reported success rate, I would check what it measures and how exceptions are classified.
What it actually proves
A clearly defined success rate shows how often an agent met the expected criteria within a particular test set. It provides performance evidence, but little detail about individual failure modes.
Princeton University’s AI Agent Reliability Tracker examines consistency, predictability, robustness, and safety alongside task performance. OpenAI’s Trace Grading documentation describes evaluating execution traces to identify problems with tool use and workflow behavior.
For the opening example, I would investigate the two exceptions separately.
The API timeout requires confirmation that the task stopped safely and left the database unchanged. The unauthorized update requires examining the approval decision, retry sequence, tool calls, and final database state.
Even one occurrence of the second failure may affect the evidence supporting that write path. Its significance cannot be determined from the overall percentage alone.
What it does not prove
Exception logs are not a complete record of undesirable behavior.
An application error, a failed test, and a control violation are different things. An agent may return a successful result while performing duplicate writes or making an unauthorized tool call. Unless those behaviors are detected, they may never appear in ordinary error logs.
As discussed in AI Audit Logs: What Events Are Missing?, retaining logs does not establish that all relevant events were captured.
I would therefore check which events the logging and evaluation mechanisms can detect, and whether execution traces and final-state records can reveal deviations that the application did not flag.
Weak-answer pattern
Two responses deserve closer examination:
“The agent achieved 98% success. The remaining failures were isolated exceptions.”
“We checked the error logs and found nothing serious.”
Both may be accurate. Neither explains the consequences of those exceptions or establishes whether the detection mechanisms can identify control failures.
I would rather examine the significant exceptions against their expected behaviour, actual tool calls, control events, and final system states.
This is particularly relevant to the exception scenarios discussed in A Successful Demo Is Not an Acceptance Test, where a successful normal workflow provides limited evidence about failure handling.
Evidence request
Please provide the test scope, scoring criteria, number of runs, and exception classifications supporting the reported success rate.
For exceptions involving approvals, permissions, tool execution, or data modification, provide the relevant execution traces, control events, retry sequences, and final system states. Identify which exceptions have been investigated or retested and which remain unresolved.
The aggregate results should remain available, but significant exceptions need enough supporting detail for separate examination.
Review note
The available results indicate a high overall success rate within the defined test set. However, some exceptional executions lack complete execution records or final-state evidence. Their consequences remain insufficiently established, and the aggregate success rate alone does not support broader reliability conclusions for the affected high-impact operations.
Usage boundary
For an agent producing internal drafts, occasional output errors may be manageable through human review. The consequences are different when an agent can modify customer records or operate privileged tools.
An unauthorized action warrants investigation even if it appears only once. If an approval-related exception remains unexplained, I would record the evidence gap and identify what needs to be verified before relying on the affected execution path.
That may mean restricting a particular tool or operation while the rest of the tested workflow remains available. The appropriate boundary depends on the action involved, the available controls, and what the execution records establish.
Success rates still have a place in acceptance reporting. They help reviewers understand overall performance and compare results across versions. But when reviewing consequential actions, I would spend more time examining executions that departed from expectations, especially those where the task appeared successful while a required control may have been bypassed.
Boundary
This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.