AI Vendor Evidence Gap Notes #12
When I look at AI agent demos now, I pay more attention to a simple question: what does a successful demo actually prove?
A demonstration shows that the system can perform a task under a particular set of conditions. Acceptance goes further. It asks whether the system has been tested against the conditions that matter for the intended use, including what should happen when the normal path fails.
That distinction becomes important quickly. An agent receives a task, calls the expected tool, pauses for the required approval, and completes the workflow. Everyone can see that it works, and it is natural to say that the agent has been tested. The demo does provide useful evidence, but it supports a narrower conclusion: this execution path worked under these conditions.
Claim
A typical statement is:
“We tested the agent and the demo worked successfully.”
If the demo was recorded, we may already know quite a lot. The agent understood the request, the relevant tool was available, permissions worked, the API responded, the normal approval path was triggered, and the expected result was produced.
That is real evidence. The gap appears when “this path worked” becomes “the agent has been tested,” because the second statement suggests a broader range of behaviour than the demo actually covered.
Why it sounds sufficient
A demo is persuasive because the system is visibly doing the work. Instead of reading architecture diagrams or documentation, we can watch the request move through tool calls, approvals, and the final action.
The problem is that demos usually use cooperative conditions. Inputs are reasonable, permissions are correct, dependencies are online, and APIs return something the workflow knows how to handle. Production is less predictable. An approval service may be unavailable, a tool call may fail, an API may return an unexpected result, or the user may phrase the task differently.
For an agent, this matters because a failed step may change what happens next. The agent may retry, select another tool, or follow another execution path. If that happens, I would want to know whether the same approval and control requirements still apply.
A happy-path demo normally does not answer that question because it was not designed to.
What it actually proves
A successful run can show that a particular version of the agent can complete a workflow under a known combination of input, configuration, permissions, tools, and dependencies. That is considerably stronger evidence than documentation alone.
Acceptance testing adds another requirement: the relevant conditions have to be identified in advance, with some expectation of how the system should behave under each of them.
For example, if an action requires human approval, I would not only record one successful approval. I would also want to know what happens when approval is rejected, when the approval service is unavailable, or when a failed tool call causes the agent to retry through another path.
Not everything needs to be tested at once. The important thing is to distinguish behaviour we have actually observed from behaviour we currently assume will occur.
A public evidence example
The distinction between documented controls and tested behaviour is visible even in public vendor evidence. The Trust Signal profile for Decagon identifies public security, AI data-usage, audit-log and guardrail surfaces, while explicitly warning that those sources do not verify implementation or control operating effectiveness.
That is the same boundary this article is concerned with. Documentation can show that a control or capability is described. To establish what happened during a particular workflow, the reviewer needs execution or test evidence.
The AI Verify Foundation report on testing real-world GenAI systems makes a related point from the testing side. Its pilot found that test design depended heavily on the use case, that teams often added adversarial or simulation testing for edge cases, and that application-level testing needs to look beyond generic model benchmarks.
Neither source defines a universal acceptance test. They do show why evidence has to move beyond documentation and a successful normal run when the question is whether a system can be relied upon under a wider operating range.
What it does not prove
One successful run does not establish repeatability. Another run may take a different execution path, particularly when the agent can choose tools or steps dynamically. The important issue is not whether every run is identical, but whether required controls remain present when the path changes.
It also tells us little about failure behaviour. A successful normal path does not show how the system responds to API timeouts, permission failures, missing data, unavailable dependencies, or rejected approvals.
Versioning creates another boundary. A test result may depend on a particular model version, prompt, agent configuration, tool set, and permission structure. When one of those changes, some previous evidence may still be useful, but its continued applicability has to be considered rather than assumed.
The demo is therefore not weak evidence. Its scope is simply narrower than “tested successfully” often implies.
Weak-answer pattern
When I see:
“The agent has been tested successfully.”
I want to know what “tested” means.
It might describe one happy-path demo. It might also describe repeated runs, approval-rejection tests, dependency failures, and retained execution records. Those represent very different evidence even though the summary sentence is identical.
The weak-answer pattern is therefore not that the statement is false. It is that the statement removes the test conditions, making it difficult for a reviewer to see what has actually been covered.
Evidence request
For an agent moving from demo toward real use, I would keep the evidence request fairly small. For important workflows, record the test condition, expected behaviour, actual result, agent or model version, tools, permissions, and relevant approval, retry, or error events.
For higher-impact actions, add a few conditions outside the happy path: approval rejection, dependency failure, tool-call failure, retry, and an attempted prohibited action.
A small matrix is often enough to separate what has actually been tested from what is still assumed.
A practical evidence request could be:
Provide the acceptance scenarios already executed for this workflow, including expected and actual behaviour for normal execution, approval rejection, dependency failure, retry, and prohibited-action conditions. Include the relevant execution records and identify the agent/model version, tools, permissions, and configuration used during testing.
This does not require a large testing framework. A few clearly defined conditions with retained evidence are more useful than a long test list with no clear connection to the behaviour being relied upon.
Review note
If the available evidence is mainly a successful demo, I would record:
The current evidence shows that the agent can complete the intended workflow under the tested conditions. It supports this execution path, but does not yet establish whether relevant controls remain consistent across repeated runs, dependency failures, retries, unexpected inputs, or alternate execution paths.
That records what the demo has established without extending the conclusion beyond the available evidence.
Usage boundary
Until those additional conditions have been tested, I would keep the agent within a scope where unexpected behaviour can still be observed and corrected before it creates a material outcome.
This matters more when an agent can modify data, send external messages, call other systems, or directly trigger business actions. In those cases, it is not enough to know that the agent can finish the task. We also need evidence of what it does when the expected workflow stops working.
A successful demo is therefore a useful starting point for acceptance work. It tells us that a workflow can run. The next step is to make the tested boundary explicit: which conditions have been exercised, what behaviour was observed, and which important conditions remain untested.
That is the practical difference between demonstration evidence and acceptance evidence.
Boundary
This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.