AI Vendor Evidence Gap Notes #14
In A Successful Demo Is Not an Acceptance Test, I discussed why one successful execution cannot establish the full acceptance boundary of an AI agent. Even after an acceptance scenario has been defined and passed, another question remains: how consistently will the agent behave when the same task is attempted again?
A passing trial gives us one observation. Repeated runs help us understand whether the expected outcome and required controls remain consistent across comparable attempts, particularly when the agent may choose different execution paths.
Claim
A common statement is:
“We ran the acceptance test and it passed.”
If the test conditions, expected behavior, and results were recorded, this is useful evidence. It shows that the agent satisfied the acceptance condition during that particular trial.
The problem arises when one successful trial is used to support a broader reliability claim. An agent may be capable of completing a task without completing it consistently enough for the intended use. Establishing that consistency requires evidence from more than one run.
Why it sounds sufficient
Traditional software testing often gives us a relatively predictable relationship between inputs and outputs. We execute a test, compare the result against an expected outcome, and investigate when the result differs.
AI agents introduce additional variation. The same request may lead to different tool choices, execution sequences, or recovery paths. External dependencies, retrieval results, and system state can also affect how a task unfolds.
Variation itself is not necessarily a failure. Two runs may follow different paths and still produce acceptable outcomes. What matters is whether the requirements we depend on remain satisfied.
Consider an agent that updates customer records. It may use different intermediate steps, but each successful run should leave the record in the expected state. If human approval is mandatory, that requirement should remain in place regardless of the path the agent selects.
One passing test shows that these conditions were satisfied once. It does not establish how consistently they will be satisfied.
What it actually proves
Repeated trials provide evidence about behavior across multiple executions under comparable conditions. They can expose differences in final outcomes, control enforcement, tool usage, retries, and exception handling that would be invisible in a single run.
I would distinguish two things when reviewing the results: outcome consistency and control consistency.
Outcome consistency concerns whether the task reaches the expected final state without unintended side effects. Control consistency concerns whether required safeguards remain effective even when the execution path changes.
This is where execution traces become important. In AI Audit Logs: What Events Are Missing?, I discussed the events needed to reconstruct an AI workflow. The same principle applies here. Without records of tool calls, approvals, retries, failures, and final outcomes, a reviewer may know that several runs succeeded without understanding how they differed.
A result such as “9 of 10 runs passed” is useful, but the failed run needs examination. A formatting error and an approval bypass have very different implications, even if both appear as one failure in the summary.
A public evidence example
Anthropic discusses this distinction in its guidance on evaluating AI agents. It treats each attempt as a separate trial and distinguishes between pass@k, which measures whether at least one of several attempts succeeds, and pass^k, which measures whether all of them succeed.
The earlier τ-bench research introduced pass^k to examine agent reliability across repeated trials. Its findings illustrate how reasonable single-run success can coexist with substantially weaker consistency when an agent must succeed repeatedly.
These measures do not replace use-case-specific acceptance criteria. They illustrate why capability and consistency need separate evidence, particularly when the system is expected to perform reliably without repeated human intervention.
What it does not prove
Repeated runs still have limitations. Executing the same happy-path scenario twenty times does not replace testing approval rejection, dependency failures, prohibited actions, or other relevant conditions. Repeatability and scenario coverage answer different questions.
The test environment also matters. If trials inherit state from previous executions or use changing dependencies, differences between runs become harder to interpret. Starting conditions, permissions, configuration, and evaluation criteria should therefore be recorded and kept comparable where practical.
I would avoid universal rules such as requiring every agent to pass ten consecutive trials. The appropriate testing depth depends on the consequences of failure, observed variation, and the intended usage boundary.
Repeated testing provides evidence within the conditions examined. It does not guarantee reliability across untested situations or after the system changes.
Weak-answer pattern
A statement such as:
“We tested it several times and it worked consistently.”
sounds reassuring, but it leaves important details unresolved.
How many trials were performed? Were the starting conditions comparable? Were failed attempts retained? Did the team evaluate only the final output, or did it also examine required controls and intermediate actions?
Three successful demonstrations selected from a larger set of attempts are not equivalent to thirty recorded trials that include failures, execution traces, and final-state verification.
The weakest pattern is reporting only the best run. That removes precisely the variability repeated testing is supposed to reveal.
Evidence request
For important workflows, I would request repeated execution records for a small number of defined acceptance scenarios.
Provide the results of repeated trials for selected acceptance scenarios, including the number of runs, starting conditions, expected outcome, actual final state, relevant tool and control events, retries, exceptions, and failed or divergent executions. Retain all trial results rather than only successful examples.
Where a control is mandatory, the evidence should show whether it remained effective across the relevant trials. A prohibited action or skipped approval should remain visible even when the overall task success rate is high.
A small set of clearly defined scenarios with complete execution evidence is more useful than a large collection of poorly specified tests.
Review note
If repeated testing has been performed but the evidence remains limited, I would record:
The available evidence shows successful execution across the recorded trials under the tested conditions. This provides some evidence of repeatability, but does not establish behaviour outside the observed sample. Control consistency and divergent execution paths require separate examination before broader reliability conclusions can be supported.
The note recognizes the successful trials while keeping the limits of that evidence visible.
Usage boundary
The amount of repeatability evidence needed depends on what the agent is allowed to do.
For an assistant producing drafts subject to human review, some variability may be manageable. For an agent modifying production records, sending external communications, or operating through privileged tools, an infrequent inconsistent execution may have significant consequences.
In those cases, I would keep the operating scope aligned with the conditions actually tested, the variability observed, and the ability to detect or contain an unexpected outcome.
Repeated testing does not make an agent deterministic. Its value is making variation observable so that reliability claims can be supported by more than a successful example.
A single run demonstrates what an agent can achieve under one set of conditions. Repeated runs provide a clearer picture of how consistently it achieves the outcomes and respects the controls we intend to rely on.
Boundary
This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.