An AI agent has completed acceptance testing. Its job is to update customer records, but every write operation requires human approval. The test report covers both approved and rejected requests, with execution traces showing whether the update was performed or blocked.

A few weeks later, the team upgrades the underlying model. The approval service and tool permissions remain unchanged, so the development team considers the previous acceptance report still valid.

That may be a reasonable position. Before relying on the report for the updated agent, though, I would want to see whether the model change affected any execution paths that the original tests were meant to cover.

Claim

“We only upgraded the model. The approval logic hasn’t changed, so we don’t need to repeat acceptance testing.”

There is some logic behind this. Software teams routinely use change impact analysis to decide which regression tests need to run. A minor component update should not automatically require the entire system to be tested again.

The difficulty with AI agents is that execution paths are not always fixed. Even when tool definitions and permissions remain unchanged, a different model may select tools differently, reorder steps, or handle failed calls in ways the previous version did not.

Why it sounds sufficient

Suppose approval is enforced by a separate backend service and every write request must pass through it. A model upgrade should not directly change that approval rule.

In that case, some of the earlier control evidence may remain applicable. The relevant question is whether the updated agent still operates within the same enforced boundary.

For example, the original agent might use one API to update a record, while the new version may choose an alternative tool after an API failure. If that alternative path was already available but never exercised during the original tests, the unchanged approval configuration tells us little about how the new execution behaves.

What it actually proves

The original acceptance report still has value. It establishes what happened under the previous model, configuration, and test conditions. If the recorded traces show that a rejected approval prevented a write operation, upgrading the model does not invalidate that historical observation.

The question is how much of that evidence still applies.

I would start with the differences between the two versions: model, system prompt, available tools, permissions, and relevant workflow configuration. Then I would compare those differences against the original acceptance scenarios.

In A Successful Demo Is Not an Acceptance Test, I discussed why acceptance evidence needs to cover the conditions that matter beyond the normal execution path. The same principle applies when deciding what needs to be tested again.

For the customer-record example, I would pay particular attention to approval rejection and API failure handling. A successful update through the normal path shows that the agent can still complete the task, but it says little about whether approval remains enforced during retries or alternate tool calls.

Where an independent backend control demonstrably covers every available write path, some existing evidence may remain sufficient. That conclusion should be supported by the relevant configuration and execution-path evidence rather than assumed from the absence of an intentional control change.

What it does not prove

Running a few successful workflows after the upgrade does not necessarily establish that regression testing covered the effects of the change.

Anthropic discusses this distinction in Demystifying Evals for AI Agents. Its discussion of regression evals focuses on whether agents continue to perform tasks they could previously handle. OpenAI’s Evaluate Agent Workflows also describes using execution traces and grading criteria to examine tool selection, handoffs, and workflow regressions.

These approaches are useful for organizing tests, but the resulting evidence still needs to correspond to the system being reviewed.

An updated agent might complete more tasks successfully while introducing a control problem in a less common execution path. Comparing overall success rates would not necessarily reveal that.

The execution record matters here. As discussed in AI Audit Logs: What Events Are Missing?, logging capability alone does not establish that the events needed to reconstruct a workflow are available. For regression testing, reviewers need enough detail to understand whether the expected approval, tool, retry, and final-state events occurred.

I would not require full retesting after every model change. I would first identify which previously tested assumptions could have been affected, then select the scenarios that need fresh evidence.

Weak-answer pattern

Two responses tend to appear together in this kind of review:

“The approval logic wasn’t modified.”

“We ran the tests again after the upgrade, and everything passed.”

Both statements may be accurate. Neither tells me enough about the evidence on its own.

The first needs supporting configuration and execution-path information. The second needs the test scope and results. Without those records, it is difficult to determine whether the updated agent introduced an alternative path that the regression tests did not exercise.

I would rather see a short explanation of which acceptance scenarios were affected, which were retested, and why the remaining evidence was considered applicable.

Evidence request

For an agent that has already passed acceptance testing and subsequently changed, I would request:

Provide the differences in model version, prompt, tool definitions, permissions, and relevant workflow configuration. Identify which previously tested behaviours and control paths may be affected.

For affected scenarios, provide regression test results and execution traces, including applicable approval rejection, tool failure, and retry conditions. For scenarios not retested, explain the basis and limitations for continuing to rely on the original evidence.

The purpose is to connect the change to the testing decision, rather than request another complete test cycle by default.

Review note

Where some regression testing has been completed but coverage remains limited, I would record:

The original acceptance evidence supports the observed behaviour of the previous system version under its recorded test conditions. Regression testing following the model change covers selected affected scenarios, but execution evidence for certain alternate paths and exception conditions remains unavailable. The current evidence does not establish whether those paths continue to satisfy the original approval requirements.

This preserves the earlier testing result while identifying the part of the current system that remains unverified.

Usage boundary

The importance of that gap depends on what the agent is allowed to do.

For an agent that produces recommendations subject to further human review, some behavioural variation may be manageable within the existing workflow. An agent with production write access creates a different concern, particularly if an untested retry or alternative tool path could affect business records.

If the updated agent can use a write path that has not been covered by current evidence, I would keep that path outside the evidenced operating scope until the relevant control behaviour has been checked. The review record should make clear which operations are supported by current-version evidence and which remain subject to further testing.

Over time, models, prompts, and tools will continue to change. An acceptance report therefore needs to remain associated with a particular system version, configuration, test environment, and evidence date.

Otherwise, the report may still be complete and historically accurate, while nobody reviewing it later can confidently establish which version of the agent it describes.

Boundary

This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

Later, someone discovers that the normalization rule was wrong.

The team opens the stored JSON, changes the affected field, and calculates a new SHA256 hash.

The new hash matches the updated file.

But what happened to the evidence that produced the original assessment?

If the previous version was overwritten, the historical policy input may no longer exist. The current record looks consistent, but the earlier conclusion cannot be replayed from that record.

A corrected evidence file is not a preserved evidence history.

Correction is a separate event

Consider a source response containing:

status = not_configured

An earlier normalization rule produces:

enabled = false

A later rule preserves the same source value as:

enabled = null, state = unknown

The original observation has not changed in this scenario.

The interpretation has.

If the first normalized record has already been referenced by a policy evaluation, replacing its contents creates a second problem: the record no longer represents what the evaluator originally consumed.

This is not a new observation. It is not target-system remediation.

It is a revision of the evidence representation.

That distinction should survive in the evidence pipeline.

Keep both versions identifiable

A safer structure preserves three separate objects: the original evidence version, the revised version, and the event connecting them.

The revision event needs enough information to establish lineage:

• Original evidence reference and digest

• Successor evidence reference and digest

• Original raw observation reference

• Revision reason and claimed actor

• Affected historical evaluation references

• Authorization and re-evaluation status

The original evaluation must remain bound to its original input.

The revised representation gets a different identity.

This is consistent with the revision relationship described by W3C PROV-O’s prov:wasRevisionOf: a revised entity can be linked to an earlier entity.

The CodeYourCompliance example does not implement the W3C ontology. The relationship is a useful conceptual reference.

But recording the relationship is not enough to establish that the correction is valid.

A revision event can still be unauthorized. Its reason can be wrong. The replacement fact may require independent review.

The lineage records what changed. It does not approve the change.

Historical replay and reassessment are different operations

As discussed in Can Your Audit Evidence Survive Replay?, replay depends on preserving the evidence and evaluation context used for the original decision.

Suppose the original assessment used evidence version v1.

A later correction produces v2.

There are now two questions.

Historical replay: What result did the original policy, using the original evidence and evaluation context, produce?

Reassessment: What result would a separate evaluation produce using the revised evidence?

The second operation must not overwrite the first.

If v1 disappears, a future reviewer may be unable to reproduce the historical assessment, even if v2 is internally consistent and its hash verifies.

An old digest without the old evidence bytes does not reconstruct the original policy input.

If another independently preserved copy exists, replay may still be possible. That is a different storage and provenance claim.

What the GitHub artifact models

This week’s Evidence Revision Lineage Contract compares two synthetic structures.

In the in-place replacement example, a normalized record is overwritten. An earlier digest reference remains, but the original payload is unavailable from the modeled package.

In the linked revision example, separate versions preserve their identities, the common raw observation reference, and the original evaluation binding. A proposed revision event connects them.

The second structure preserves more information.

It does not prove the correction was approved, that either version is correct, or that an append-only storage system exists.

These are static synthetic examples. They do not execute a collector, calculate real hashes, enforce immutable storage, or replay an OPA policy. The original evaluation reference is modeled, not a runtime capture.

That is the current implementation boundary.

The audit problem starts before correction

Evidence pipelines need a way to handle errors discovered after collection.

A parser may be corrected. A source reference may be challenged. A previous evidence assertion may need to be withdrawn.

None of these events should silently mutate the historical inputs.

The pipeline should distinguish observation, transformation, revision, policy evaluation, and remediation. Each has a different provenance and authority boundary.

As argued in Compliance Automation Starts at Evidence, automating the report is not the same as establishing the evidence behind it.

The same applies to corrections. A clean, updated report does not establish what was originally observed or evaluated.

For MAS TRM-inspired engineering, this is an interpretation of what evidence must preserve to support later review. It is not a claim that the MAS Technology Risk Management Guidelines prescribe this revision schema or implementation mechanism.

A new evidence version may change what can be concluded today.

It must not erase what was actually evaluated yesterday.

Public reference artifacts

The accompanying technical material is available in the CodeYourCompliance evidence-validation-pipeline repository:

• Evidence Revision Lineage Contract

• Synthetic Revision Comparison

• Current Implementation Status

Origin and scope

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

CodeYourCompliance explores replayable evidence, read-only evidence collection, provenance, integrity verification, and policy-as-code.

Attribution is requested for forks, references, adaptations, and technical discussions.

MAS TRM-inspired means engineering interpretation, not legal, regulatory, audit, certification, or compliance advice.

In A Successful Demo Is Not an Acceptance Test, I discussed why one successful execution cannot establish the full acceptance boundary of an AI agent. Even after an acceptance scenario has been defined and passed, another question remains: how consistently will the agent behave when the same task is attempted again?

A passing trial gives us one observation. Repeated runs help us understand whether the expected outcome and required controls remain consistent across comparable attempts, particularly when the agent may choose different execution paths.

Claim

A common statement is:

“We ran the acceptance test and it passed.”

If the test conditions, expected behavior, and results were recorded, this is useful evidence. It shows that the agent satisfied the acceptance condition during that particular trial.

The problem arises when one successful trial is used to support a broader reliability claim. An agent may be capable of completing a task without completing it consistently enough for the intended use. Establishing that consistency requires evidence from more than one run.

Why it sounds sufficient

Traditional software testing often gives us a relatively predictable relationship between inputs and outputs. We execute a test, compare the result against an expected outcome, and investigate when the result differs.

AI agents introduce additional variation. The same request may lead to different tool choices, execution sequences, or recovery paths. External dependencies, retrieval results, and system state can also affect how a task unfolds.

Variation itself is not necessarily a failure. Two runs may follow different paths and still produce acceptable outcomes. What matters is whether the requirements we depend on remain satisfied.

Consider an agent that updates customer records. It may use different intermediate steps, but each successful run should leave the record in the expected state. If human approval is mandatory, that requirement should remain in place regardless of the path the agent selects.

One passing test shows that these conditions were satisfied once. It does not establish how consistently they will be satisfied.

What it actually proves

Repeated trials provide evidence about behavior across multiple executions under comparable conditions. They can expose differences in final outcomes, control enforcement, tool usage, retries, and exception handling that would be invisible in a single run.

I would distinguish two things when reviewing the results: outcome consistency and control consistency.

Outcome consistency concerns whether the task reaches the expected final state without unintended side effects. Control consistency concerns whether required safeguards remain effective even when the execution path changes.

This is where execution traces become important. In AI Audit Logs: What Events Are Missing?, I discussed the events needed to reconstruct an AI workflow. The same principle applies here. Without records of tool calls, approvals, retries, failures, and final outcomes, a reviewer may know that several runs succeeded without understanding how they differed.

A result such as “9 of 10 runs passed” is useful, but the failed run needs examination. A formatting error and an approval bypass have very different implications, even if both appear as one failure in the summary.

A public evidence example

Anthropic discusses this distinction in its guidance on evaluating AI agents. It treats each attempt as a separate trial and distinguishes between pass@k, which measures whether at least one of several attempts succeeds, and pass^k, which measures whether all of them succeed.

The earlier τ-bench research introduced pass^k to examine agent reliability across repeated trials. Its findings illustrate how reasonable single-run success can coexist with substantially weaker consistency when an agent must succeed repeatedly.

These measures do not replace use-case-specific acceptance criteria. They illustrate why capability and consistency need separate evidence, particularly when the system is expected to perform reliably without repeated human intervention.

What it does not prove

Repeated runs still have limitations. Executing the same happy-path scenario twenty times does not replace testing approval rejection, dependency failures, prohibited actions, or other relevant conditions. Repeatability and scenario coverage answer different questions.

The test environment also matters. If trials inherit state from previous executions or use changing dependencies, differences between runs become harder to interpret. Starting conditions, permissions, configuration, and evaluation criteria should therefore be recorded and kept comparable where practical.

I would avoid universal rules such as requiring every agent to pass ten consecutive trials. The appropriate testing depth depends on the consequences of failure, observed variation, and the intended usage boundary.

Repeated testing provides evidence within the conditions examined. It does not guarantee reliability across untested situations or after the system changes.

Weak-answer pattern

A statement such as:

“We tested it several times and it worked consistently.”

sounds reassuring, but it leaves important details unresolved.

How many trials were performed? Were the starting conditions comparable? Were failed attempts retained? Did the team evaluate only the final output, or did it also examine required controls and intermediate actions?

Three successful demonstrations selected from a larger set of attempts are not equivalent to thirty recorded trials that include failures, execution traces, and final-state verification.

The weakest pattern is reporting only the best run. That removes precisely the variability repeated testing is supposed to reveal.

Evidence request

For important workflows, I would request repeated execution records for a small number of defined acceptance scenarios.

Provide the results of repeated trials for selected acceptance scenarios, including the number of runs, starting conditions, expected outcome, actual final state, relevant tool and control events, retries, exceptions, and failed or divergent executions. Retain all trial results rather than only successful examples.

Where a control is mandatory, the evidence should show whether it remained effective across the relevant trials. A prohibited action or skipped approval should remain visible even when the overall task success rate is high.

A small set of clearly defined scenarios with complete execution evidence is more useful than a large collection of poorly specified tests.

Review note

If repeated testing has been performed but the evidence remains limited, I would record:

The available evidence shows successful execution across the recorded trials under the tested conditions. This provides some evidence of repeatability, but does not establish behaviour outside the observed sample. Control consistency and divergent execution paths require separate examination before broader reliability conclusions can be supported.

The note recognizes the successful trials while keeping the limits of that evidence visible.

Usage boundary

The amount of repeatability evidence needed depends on what the agent is allowed to do.

For an assistant producing drafts subject to human review, some variability may be manageable. For an agent modifying production records, sending external communications, or operating through privileged tools, an infrequent inconsistent execution may have significant consequences.

In those cases, I would keep the operating scope aligned with the conditions actually tested, the variability observed, and the ability to detect or contain an unexpected outcome.

Repeated testing does not make an agent deterministic. Its value is making variation observable so that reliability claims can be supported by more than a successful example.

A single run demonstrates what an agent can achieve under one set of conditions. Repeated runs provide a clearer picture of how consistently it achieves the outcomes and respects the controls we intend to rely on.

Boundary

This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The sender says the file came from a production system. The reviewer calculates the hash again, and it matches.

Can we now say the evidence is authentic?

Not yet.

The file and its hash may have come from the same sender. Someone could have modified the original content, generated a new hash, and delivered both together. The hash would still match.

The calculation is correct. But we have not established where the evidence originally came from.

What a matching hash actually proves

A SHA256 comparison answers a specific question:

computed_hash(received_bytes)
==
reference_hash

If they match, the received bytes correspond to the reference hash used in that comparison.

But where did the reference hash come from?

If it was independently captured and protected during collection, that gives us a stronger basis for detecting later changes.

If it was supplied alongside the file by the same unverified sender, the comparison cannot establish that the file is unchanged from the original collection.

This is a different problem from the one SHA256 was designed to solve.

And it becomes more interesting when we look at the evidence metadata.

A collector name is still a claim

An evidence object may already contain:

collector:
  name: ansible-readonly-collector
  version: 0.1.0

source:
  system: production-server-01

These fields are useful. They tell us which collector and source the evidence claims to represent.

But they do not independently authenticate that collector.

A field containing production-server-01 does not prove that the bytes came from that server. Similarly, a collector version does not prove which executable actually ran.

This is why collector provenance and evidence authenticity need to remain separate.

Collector metadata helps us reconstruct the claimed production path. Authenticity requires additional assurance that the claimed producer is genuinely bound to the evidence.

Even an authenticated producer does not automatically prove that the source was authoritative or that the collected facts were correct.

A recorded handoff is not a verified custody chain

Now suppose the evidence pipeline also records a handoff:

custody_event:
  from: collector
  to: evidence_repository
  subject_sha256: ...
  received_at_utc: ...

This is useful information. It makes the claimed transfer visible.

But who recorded the event? Were the two parties authenticated? Was the evidence digest checked against an independently protected reference? And what happened before and after this handoff?

A JSON record containing those fields does not answer these questions by itself.

The term chain of custody is used in digital evidence handling to describe tracking evidence through collection, handling, and transfer. That includes recording who handled the evidence, when, and for what purpose.

For an automated evidence pipeline, the engineering lesson is narrower: we need to distinguish a recorded custody event from an authenticated and complete custody history.

One recorded handoff does not establish the entire chain.

Two synthetic scenarios

The latest CodeYourCompliance artifact models this distinction using two scenarios involving the same synthetic evidence subject.

The first scenario contains a modeled digest correspondence and claimed collector metadata, but no handoff event.

modeled_digest_match = true
integrity_verification = not_executed
origin_authentication = not_established
handoff_count = 0

The second scenario uses the same evidence subject but adds one recorded handoff.

modeled_digest_match = true
integrity_verification = not_executed
origin_authentication = not_established
handoff_count = 1
custody_chain = not_assessed

The second record contains more information than the first.

But it still does not establish authenticated origin or a verified custody chain.

There is another important limitation: these are synthetic models. No real hash verification, transfer, signature verification, or policy evaluation was executed. Even the digest is explicitly marked as a placeholder.

The examples are intended to expose the distinction, not demonstrate a finished custody-verification system.

Where authenticity belongs in the evidence pipeline

Previous CodeYourCompliance work has separated collector provenance, transformation provenance, policy evaluation, observation outcomes, and time provenance.

Custody and authenticity add another boundary.

Before an evidence object is accepted for policy evaluation, we may need to know not only whether its bytes match a digest, but also what assurance exists for its claimed origin and handling history.

That does not mean every evidence package must immediately use digital signatures or an immutable evidence store.

It means the pipeline should preserve what it actually knows.

If the reference digest is self-supplied, record that.

If a handoff is only asserted, record it as an asserted handoff.

If origin authentication was never performed, keep that state as not_established.

Otherwise a series of unverified metadata fields can gradually become a report that describes the evidence as fully authenticated.

The report looks stronger. The underlying evidence has not changed.

MAS TRM-inspired means engineering interpretation

This is not a claim that the MAS Technology Risk Management Guidelines prescribe this custody schema, cryptographic signature mechanism, or particular evidence architecture.

The engineering interpretation is limited to preserving integrity, origin, and custody distinctions before making stronger technical control claims.

A matching hash does not authenticate the evidence producer.

A handoff record does not establish a verified chain of custody.

Those claims require their own evidence.

Public reference artifacts

The accompanying technical material is available in the CodeYourCompliance evidence-validation-pipeline repository:

• Evidence Custody and Authenticity Contract

• Integrity-Only vs Recorded-Handoff Synthetic Examples

• Current Implementation Status

Origin and scope

CodeYourCompliance

Website: Compliance Evidence Automation | CodeYourCompliance

GitHub: https://github.com/codeyourcompliance

CodeYourCompliance explores compliance automation, replayable audit evidence, read-only evidence collection, policy-as-code, and evidence packages.

Attribution is requested for forks, references, adaptations, and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification, or compliance advice.

Related reading

• The Collector Is Part of the Evidence

• Can Your Audit Evidence Survive Replay?

• A Control Result Is Not Replayable Evidence

“Human in the loop” is one of those phrases that sounds reassuring almost immediately. If an AI agent is about to do something important, a person reviews it first. The risk appears bounded because the system is not acting entirely on its own.

As I argued in A Policy Is Not Evidence That an AI Agent Obeys It, a documented control and an enforced control are different evidence questions. Human oversight is a good example of that distinction. The first is whether a human-review mechanism exists. The second is whether that mechanism actually changes what happens when the system runs.

For oversight to be meaningful, the human needs a real decision point before the relevant action takes effect, enough information to understand what is being proposed, authority to stop or change it, and a system that actually respects that decision. A documented approval step may describe that design. The stronger evidence comes from seeing the decision and its effect in the execution record.

Claim

A common statement is:

“Sensitive actions require human approval before execution.”

That could describe a useful control. It tells us something about how the workflow is intended to operate, and a configuration screen may show that an approval requirement has been enabled.

What it does not yet tell us is whether a particular action was actually intercepted, what the reviewer saw, who made the decision, or whether rejection prevented the action from continuing.

Those details become important because “human approval” can refer to several different things. A person might approve an exact tool call before execution, review a proposed action at a higher level, check the result after the action has already happened, or simply be available to intervene if something looks wrong. All involve a human, but they do not provide the same control.

Why it sounds sufficient

Human oversight is easy to accept as evidence because it appears to solve the autonomy problem directly. If the agent cannot proceed without a person, then the person seems to remain in control.

A demo can reinforce that impression. The agent proposes an action, an approval dialog appears, someone clicks Approve, and the workflow continues. That is useful evidence that the approval path can work. That is the same evidence boundary discussed in A Successful Demo Is Not an Acceptance Test: one successful execution shows that a path can work, but it does not establish how the control behaves across the conditions we intend to rely on.

The harder question is what happens around that path. Does every action in scope trigger it? Does rejection actually stop the action? If the agent retries or chooses another tool, does the same requirement still apply? What happens when the approval service is unavailable?

These are runtime questions. A policy or configuration can describe the intended control, but it cannot answer them by itself.

What it actually proves

Different evidence layers support different conclusions.

A policy can establish that human approval is expected. Configuration evidence can show that a tool or workflow has been marked as requiring approval. A screenshot can show the reviewer interface. None of those is useless, but they mainly describe the design and configuration of the control.

Earlier in the series, Human Review Is a Data Exposure Path Unless It Is Bounded looked at human review mainly as a data-path question: who can see what, and under which conditions. Here the evidence question is different. If the human is being relied upon as a control, we also need evidence that the decision actually changes execution.

Runtime evidence gets closer to that question. For a specific execution, I would want to see the action that triggered review, the point at which execution paused, the decision that was made, and what happened to the action afterwards.

The useful chain is fairly simple:

proposed action → approval trigger → human decision → execution outcome

If the reviewer rejects the action, the trace should make it possible to see that the rejected action did not occur. If the reviewer approves it, the resulting action should be linked back to that approval rather than appearing as an unrelated event later in the workflow.

That is what turns the presence of a human-review mechanism into evidence that the mechanism actually affected execution.

A public evidence example

The distinction is visible in the documentation for the OpenAI Agents SDK human-in-the-loop flow. The SDK allows a tool call to be marked as requiring approval; when approval is needed, the run pauses, records an interruption, and can later resume after the pending action is approved or rejected. The documentation also explains that approval can apply across nested agent/tool execution rather than only at the top-level agent.

That documentation is useful configuration and control-design evidence. For a particular deployed application, however, I would still want the execution record showing that the relevant tool call actually paused, which pending action was presented for review, what decision was made, and whether the resulting execution state matched that decision.

There is another detail in the same documentation that illustrates why the distinction matters. Some tool types can use programmatic approval callbacks instead of waiting for a person. That is a legitimate implementation option, but it means the word “approval” alone does not establish that a human actually made the decision. The execution evidence needs to distinguish the mechanism used in the workflow being reviewed.

A similar boundary appears in the Trust Signal profile for Decagon, where audit logs and AI guardrails are recorded as observed enterprise-readiness evidence, while the profile explicitly avoids treating those public surfaces as proof that implementation matches the documentation.

That is the evidence gap here: knowing that an oversight capability exists is different from having evidence that it governed the execution being relied upon.

What it does not prove

Even a recorded approval event leaves some questions open.

The reviewer may have been shown too little information to make a meaningful decision. An approval screen that says “Allow tool call?” is not equivalent to showing the target system, proposed parameters, affected data, and expected consequence. The evidence should therefore include enough context to understand what the person was actually approving.

Identity and authority matter as well. If the control is supposed to require approval from a particular role, a generic approval event does not establish that the correct person made the decision. The reviewer identity or role needs to be linked to the event in a way that can be checked.

There is also the possibility of another execution path. An agent may receive a rejection, retry with different parameters, invoke another tool, or reach the same business outcome through a different workflow. The original rejection is evidence that one action was blocked; it does not automatically establish that the restricted outcome remained blocked afterwards.

Finally, the failure mode matters. If the approval service is unavailable, the system may stop, wait, time out, escalate, or continue through another path. Until that behavior has been observed or tested, “human approval required” still leaves an important part of the operating boundary unresolved.

Weak-answer pattern

A statement such as:

“A human reviews all high-risk actions.”

may be entirely accurate as a policy statement, but it becomes a weak answer when there is no evidence showing how review is triggered and what effect the decision has on execution.

I would want to know whether “review” means pre-action approval, post-action review, exception handling, or general supervision. I would also want to see at least some completed approval and rejection events rather than relying only on the configured workflow.

The gap is not that human oversight is absent. The gap is that its operating effect has not yet been evidenced.

Evidence request

For a workflow that relies on human approval as a control, I would ask for a small sample of completed oversight events rather than a large policy package.

A practical evidence request could be:

Provide execution records for representative human-oversight events, including the action that triggered review, the information presented to the reviewer, reviewer identity or role, approval or rejection decision, timestamp, linked tool/action record, and final execution state. Include at least one rejection case and, where relevant, a case in which approval was unavailable or the agent attempted an alternate path.

This gives the reviewer something that can actually be followed from the proposed action through to the resulting system behavior.

It also avoids turning the exercise into a general assessment of whether the human made a “good” decision. The immediate evidence question is narrower: did the oversight control occur when expected, and did the system behave according to that decision?

Review note

If the current evidence consisted mainly of a policy, configuration, or demo showing the approval mechanism, I would record something like:

Current evidence shows that a human-approval mechanism is defined and can be invoked for the reviewed workflow. The available material does not yet establish that all in-scope actions consistently trigger the control, that rejection prevents equivalent execution through alternate paths, or that the resulting action can be reliably linked to the recorded approval decision.

This preserves the value of the evidence already available without treating control design as evidence of operating effectiveness.

Usage boundary

Until the runtime behavior of the oversight mechanism has been demonstrated, I would keep the agent within a scope where an approval failure or bypass remains observable and recoverable.

That is particularly relevant for actions that modify production data, send external communications, create financial or contractual effects, change access, or trigger another system. In those cases, human oversight is being relied upon as part of the control boundary, so the evidence should show more than the existence of an Approve button.

A useful human-in-the-loop control is not simply a workflow with a person somewhere in it. The evidence should let us follow the decision from the point where intervention was required through to what the system ultimately did.

That is the point where human oversight becomes execution evidence rather than only a documented expectation.

Boundary

This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

collected_at_utc = 2026-09-22T01:00:00Z

A few seconds later, the object is hashed with SHA256.

The hash verifies.

It is tempting to read this as evidence that the system was observed at 01:00 UTC.

But the timestamp may only tell us what the collector’s clock said.

If that clock was wrong, the timestamp can be wrong. If the object was created later with a backdated timestamp and then hashed, the hash can still verify.

Nothing is wrong with SHA256.

We are asking it to prove something it does not prove.

A timestamp records a time claim. It does not by itself prove when the evidence existed.

Hash integrity is not time assurance

The useful integrity boundary is:

hash verifies
=> stored bytes match recorded digest

It is not:

hash verifies
=> embedded timestamp is historically true

A valid hash can protect a false timestamp just as effectively as a correct one.

This matters because evidence packages often put these fields next to each other:

collected_at_utc
sealed_at_utc
sha256

That structure looks strong.

But collected_at_utc may still be an assertion from the same process that created the evidence.

Integrity and time assurance are different problems.

There is more than one time in the evidence chain

An evidence pipeline may contain several different time statements:

source event time
collection time
seal time
receipt time
trusted attestation time

They answer different questions.

A source event time says when the source claims something happened.

A collection time says when the collector claims it observed the source.

A seal time says when the evidence packager claims it created the digest.

Those values can still be useful. Their provenance just needs to remain visible.

For example:

collection_time:
  value_utc: 2026-09-22T01:00:00Z
  asserted_by: synthetic-readonly-collector
  clock_source_ref: collector-host-system-clock
  assurance: collector_asserted

The important part is not only the UTC value.

It is also:

assurance = collector_asserted

An independent receipt creates a different claim

Suppose the collector claims the evidence was collected at 01:00.

A separate receiver later records the same evidence digest at 01:03.

Under the assumptions of that receiving system, this can support:

this evidence digest existed no later than 01:03

It still does not prove:

the target was observed at exactly 01:00

Those are different claims.

The receipt strengthens time provenance because the evidence crossed a separate recording boundary. But it is still not the same as trusted timestamping.

The current CodeYourCompliance model therefore keeps three states separate:

collector_asserted
independent_receipt
trusted_timestamp

They should not become different labels for the same level of assurance.

Freshness also depends on the time claim

Consider a deterministic freshness rule:

evaluation_time - collected_at <= allowed_age

OPA can calculate this correctly every time.

But if collected_at came only from the collector clock, the result still depends on that asserted time.

So two questions remain separate:

Was the freshness calculation correct?

and:

How much assurance do we have in the collection time?

A deterministic policy result answers the first.

It does not automatically answer the second.

Replay must preserve the original time semantics

The same boundary applies later during evidence replay.

If the original assessment used collector-asserted time, replay should not silently upgrade that timestamp into independently verified time simply because the evidence hash still matches.

The historical chain may have been:

observation
-> collector-asserted time
-> evidence bytes
-> integrity seal
-> policy evaluation

If an independent receipt existed, that relationship should survive replay.

If trusted timestamping existed, its attestation should remain separately bound to the evidence digest.

Replay should preserve not only the evidence bytes and policy version, but also what was actually known about the evidence at the time.

This does not mean every pipeline needs a timestamp authority

The new Evidence Time Provenance Contract deliberately stops before that conclusion.

The accompanying synthetic examples compare only:

• collector-asserted collection and seal times

• the same modeled evidence digest with a separately recorded receipt time

They do not implement trusted timestamping.

They do not validate NTP synchronization, host clock correctness, receiver authentication, or RFC 3161 Time-Stamp Protocol timestamp tokens.

The boundary is narrower.

If the system only has a collector-asserted timestamp, record it as collector-asserted.

If another system creates a separate time boundary, preserve that relationship.

If trusted timestamping does not exist, do not imply that it does.

MAS TRM-inspired means engineering interpretation

This is not a claim that the MAS Technology Risk Management Guidelines prescribe this time-provenance structure or any particular timestamping mechanism.

The engineering interpretation is simpler.

If evidence supports a point-in-time technical claim, the pipeline should preserve not only the timestamp value but also where that time came from and what assurance it provides.

Otherwise:

2026-09-22T01:00:00Z

looks precise while hiding the more important question:

Who said it was 01:00?

Public reference artifacts

• Evidence Time Provenance Contract

• Evidence Time Provenance Synthetic Examples

• Current Implementation Status

Origin and scope

CodeYourCompliance

Website: Compliance Evidence Automation | CodeYourCompliance

GitHub: https://github.com/codeyourcompliance

CodeYourCompliance explores compliance automation, replayable audit evidence, read-only evidence collection, policy-as-code, and evidence packages.

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• Can Your Audit Evidence Survive Replay?

• Compliance Automation Starts at Evidence

• Invalid Evidence Is Not a Failed Control

When I look at AI agent demos now, I pay more attention to a simple question: what does a successful demo actually prove?

A demonstration shows that the system can perform a task under a particular set of conditions. Acceptance goes further. It asks whether the system has been tested against the conditions that matter for the intended use, including what should happen when the normal path fails.

That distinction becomes important quickly. An agent receives a task, calls the expected tool, pauses for the required approval, and completes the workflow. Everyone can see that it works, and it is natural to say that the agent has been tested. The demo does provide useful evidence, but it supports a narrower conclusion: this execution path worked under these conditions.

Claim

A typical statement is:

“We tested the agent and the demo worked successfully.”

If the demo was recorded, we may already know quite a lot. The agent understood the request, the relevant tool was available, permissions worked, the API responded, the normal approval path was triggered, and the expected result was produced.

That is real evidence. The gap appears when “this path worked” becomes “the agent has been tested,” because the second statement suggests a broader range of behaviour than the demo actually covered.

Why it sounds sufficient

A demo is persuasive because the system is visibly doing the work. Instead of reading architecture diagrams or documentation, we can watch the request move through tool calls, approvals, and the final action.

The problem is that demos usually use cooperative conditions. Inputs are reasonable, permissions are correct, dependencies are online, and APIs return something the workflow knows how to handle. Production is less predictable. An approval service may be unavailable, a tool call may fail, an API may return an unexpected result, or the user may phrase the task differently.

For an agent, this matters because a failed step may change what happens next. The agent may retry, select another tool, or follow another execution path. If that happens, I would want to know whether the same approval and control requirements still apply.

A happy-path demo normally does not answer that question because it was not designed to.

What it actually proves

A successful run can show that a particular version of the agent can complete a workflow under a known combination of input, configuration, permissions, tools, and dependencies. That is considerably stronger evidence than documentation alone.

Acceptance testing adds another requirement: the relevant conditions have to be identified in advance, with some expectation of how the system should behave under each of them.

For example, if an action requires human approval, I would not only record one successful approval. I would also want to know what happens when approval is rejected, when the approval service is unavailable, or when a failed tool call causes the agent to retry through another path.

Not everything needs to be tested at once. The important thing is to distinguish behaviour we have actually observed from behaviour we currently assume will occur.

A public evidence example

The distinction between documented controls and tested behaviour is visible even in public vendor evidence. The Trust Signal profile for Decagon identifies public security, AI data-usage, audit-log and guardrail surfaces, while explicitly warning that those sources do not verify implementation or control operating effectiveness.

That is the same boundary this article is concerned with. Documentation can show that a control or capability is described. To establish what happened during a particular workflow, the reviewer needs execution or test evidence.

The AI Verify Foundation report on testing real-world GenAI systems makes a related point from the testing side. Its pilot found that test design depended heavily on the use case, that teams often added adversarial or simulation testing for edge cases, and that application-level testing needs to look beyond generic model benchmarks.

Neither source defines a universal acceptance test. They do show why evidence has to move beyond documentation and a successful normal run when the question is whether a system can be relied upon under a wider operating range.

What it does not prove

One successful run does not establish repeatability. Another run may take a different execution path, particularly when the agent can choose tools or steps dynamically. The important issue is not whether every run is identical, but whether required controls remain present when the path changes.

It also tells us little about failure behaviour. A successful normal path does not show how the system responds to API timeouts, permission failures, missing data, unavailable dependencies, or rejected approvals.

Versioning creates another boundary. A test result may depend on a particular model version, prompt, agent configuration, tool set, and permission structure. When one of those changes, some previous evidence may still be useful, but its continued applicability has to be considered rather than assumed.

The demo is therefore not weak evidence. Its scope is simply narrower than “tested successfully” often implies.

Weak-answer pattern

When I see:

“The agent has been tested successfully.”

I want to know what “tested” means.

It might describe one happy-path demo. It might also describe repeated runs, approval-rejection tests, dependency failures, and retained execution records. Those represent very different evidence even though the summary sentence is identical.

The weak-answer pattern is therefore not that the statement is false. It is that the statement removes the test conditions, making it difficult for a reviewer to see what has actually been covered.

Evidence request

For an agent moving from demo toward real use, I would keep the evidence request fairly small. For important workflows, record the test condition, expected behaviour, actual result, agent or model version, tools, permissions, and relevant approval, retry, or error events.

For higher-impact actions, add a few conditions outside the happy path: approval rejection, dependency failure, tool-call failure, retry, and an attempted prohibited action.

A small matrix is often enough to separate what has actually been tested from what is still assumed.

A practical evidence request could be:

Provide the acceptance scenarios already executed for this workflow, including expected and actual behaviour for normal execution, approval rejection, dependency failure, retry, and prohibited-action conditions. Include the relevant execution records and identify the agent/model version, tools, permissions, and configuration used during testing.

This does not require a large testing framework. A few clearly defined conditions with retained evidence are more useful than a long test list with no clear connection to the behaviour being relied upon.

Review note

If the available evidence is mainly a successful demo, I would record:

The current evidence shows that the agent can complete the intended workflow under the tested conditions. It supports this execution path, but does not yet establish whether relevant controls remain consistent across repeated runs, dependency failures, retries, unexpected inputs, or alternate execution paths.

That records what the demo has established without extending the conclusion beyond the available evidence.

Usage boundary

Until those additional conditions have been tested, I would keep the agent within a scope where unexpected behaviour can still be observed and corrected before it creates a material outcome.

This matters more when an agent can modify data, send external messages, call other systems, or directly trigger business actions. In those cases, it is not enough to know that the agent can finish the task. We also need evidence of what it does when the expected workflow stops working.

A successful demo is therefore a useful starting point for acceptance work. It tells us that a workflow can run. The next step is to make the tested boundary explicit: which conditions have been exercised, what behaviour was observed, and which important conditions remain untested.

That is the practical difference between demonstration evidence and acceptance evidence.

Boundary

This material is for evidence structuring, review preparation, and evidence-oriented testing discussion. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

It is easy to normalize that result immediately as:

configured = false

Sometimes that is correct.

But sometimes the source was never successfully observed in the first place.

API timeouts, permission problems, incomplete pagination, an unstable or unreachable path between the collector and the source, or even a successful request that did not actually cover the full intended scope can all leave the pipeline with “no object returned.”

Those situations are not the same.

If their collection outcomes are collapsed too early, a failed observation can quietly become negative evidence.

Two empty results that mean different things

The latest CodeYourCompliance example models two collection attempts against the same synthetic target.

In the first scenario, we assume the collector successfully reached the intended source and scope. The query completed normally, the result set was complete, and no matching legacy TLS configuration object was found.

The modeled result is:

collection_status = success
result_completeness = complete
observed_state = observed_absent
absence_assertion_supported = true

In the second scenario, the same intended query timed out before the source state was obtained:

collection_status = failed
result_completeness = not_observed
observed_state = unobserved
absence_assertion_supported = false

Neither scenario returns a matching object.

Only the first supports an assertion that the object was absent.

They describe two different observation states, so normalization should not erase that difference.

Absence is still an observation

Negative evidence is easy to describe as “nothing was there.”

That is too loose.

To state that something was absent, the collection path still needs to show what was observed, where it was observed, how it was queried, and whether the observation actually completed.

Not seeing something and confirming its absence are different things.

A bounded absence statement might look like this:

At timestamp T,
through query X,
against synthetic-endpoint-443,
no matching legacy TLS protocol configuration was observed.

That is not the same as:

Legacy TLS is disabled.

The first statement describes an observation.

The second is already a configuration conclusion and may later become part of a control conclusion.

Moving from the first to the second may require more evidence or policy logic.

HTTP 200 is not enough

The first version of the synthetic example exposed an important problem during adversarial review.

It was too easy to read:

HTTP 200
+ empty result

as sufficient proof that the object was absent.

That is too strong.

These responses can look perfectly normal without proving that the intended observation was complete.

The revised synthetic example therefore makes several assumptions explicit. For the observed_absent scenario, we assume by construction that:

• the intended source is authoritative for the synthetic condition

• the intended scope was covered

• the query semantics are defined

• pagination is complete

• the collecting identity has sufficient visibility over the declared scope

These are scenario assumptions.

They are not runtime validations.

The example therefore does not claim that:

HTTP 200 + empty payload

generally means absence.

Collection status belongs in the evidence chain

The new Observed Absence Evidence Contract keeps collection outcome separate from the later control result.

A minimum record can include:

observation_outcome:
  source_ref:
  collection_method:
  collection_attempted:
  collection_status:
  query_semantics_ref:
  scope_ref:
  scope_reached:
  response_status:
  result_completeness:
  observed_state:
  absence_assertion_supported:
  failure_reason:
  observed_at_utc:

The point is not to preserve every transport detail.

The point is to stop materially different collection outcomes from being normalized into the same fact.

At minimum, the pipeline may need to distinguish states such as:

present
observed_absent
unobserved
indeterminate

observed_absent means the observation completed and the query semantics support an absence statement.

unobserved means the intended source state was never successfully obtained.

A timeout belongs in the second state. It should not be converted into false.

A failed observation should stay failed

Suppose an API request times out.

The collector does know one thing: the collection attempt failed.

It does not know whether the configuration object exists.

A more defensible record is therefore:

collection_status = failed
observed_state = unobserved
absence_assertion_supported = false

not:

configured = false

This becomes important once the evidence reaches policy evaluation.

If unobserved is converted into false, downstream logic can no longer distinguish between:

the condition was observed to be absent

and:

we never successfully checked it

And that is how a collection problem quietly turns into a claim about the target state.

This happens before evidence sufficiency

Last week’s control-to-evidence mapping work asked whether all required proof inputs had been assembled before policy evaluation.

This week’s problem happens earlier.

Before the pipeline can ask whether the evidence set is complete, it first has to know what each collection attempt actually produced.

The path is closer to:

source system
-> collection attempt
-> observation outcome
-> raw evidence
-> transformation
-> normalized fact
-> evidence admissibility
-> evidence sufficiency
-> policy evaluation
-> control result

An unobserved outcome may eventually leave an evidence requirement unsatisfied.

An observed_absent outcome may itself become valid evidence for a particular requirement.

But those decisions should happen after the collection outcome has been preserved, not after both states have already been flattened into the same Boolean value.

Observed absence still does not mean PASS

There is another boundary.

Even when observed_absent is recorded correctly, it does not mean the control automatically passes.

Suppose the collector confirms that no legacy TLS configuration object was observed.

Whether that absence supports the control still depends on the control design, source authority, evidence mapping, and policy semantics.

The collection layer should make only the narrower statement it can support.

It should not make the final control decision.

For that reason, the synthetic comparison stops before control evaluation.

At that point it models only the observation outcome.

Public reference artifacts

The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline repository:

• Observed Absence Evidence Contract

• Observed Absence vs Collection Failure Example

• Current Implementation Status

The current artifact models collection-outcome semantics through synthetic examples.

It does not yet validate query semantics, pagination completeness, permission visibility, or source authority automatically.

It also does not determine whether an observed absence is sufficient to satisfy a real control requirement.

Origin and scope

CodeYourCompliance

Website: Compliance Evidence Automation | CodeYourCompliance

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• The Collector Is Part of the Evidence

• Invalid Evidence Is Not a Failed Control

• A Control Result Is Not Replayable Evidence

The first is present. Its schema is valid, integrity is verified, freshness is acceptable, and the required provenance is available.

The second is missing.

A pipeline that checks only the evidence it received may still continue to control evaluation. Every object in front of it looks valid, but the proof set is incomplete.

Nothing is wrong with the first evidence object. There is simply not enough evidence to support the declared control claim.

That is a different problem from invalid evidence.

Admissibility applies to an evidence object

The evidence-admissibility gate answers whether an individual object can be used.

For example:

schema = valid
integrity = verified
freshness = fresh
provenance = present

Those checks say something about the quality of that object.

They do not establish that every evidence requirement for the control has been satisfied.

A control can therefore reach this state:

present evidence = admissible
required evidence set = incomplete
control decision = blocked
control status = unknown

The missing evidence should remain missing. It should not be converted into either a pass or a failure.

This is the boundary introduced in the latest CodeYourCompliance artifact: object-level admissibility and evidence-set sufficiency are separate checks.

A simple TLS example

The public Control-to-Evidence Mapping Example uses a synthetic TLS control.

The declared control requires two evidence types:

tls_certificate_state
tls_runtime_protocol_state

In the complete scenario, both requirements are modeled as satisfied.

The resulting state is:

evidence_set_status = complete
policy_evaluation_allowed = true
control_status = not_evaluated

The example deliberately stops before control evaluation. A complete evidence set means the downstream policy is allowed to run. It does not mean the control has passed.

The incomplete scenario contains certificate-state evidence but no runtime protocol-state evidence:

evidence_set_status = incomplete
policy_evaluation_allowed = false
control_status = unknown

The certificate evidence remains usable. The absent protocol evidence does not become a failed control condition.

The examples are synthetic. They do not execute an admissibility gate, sufficiency gate, downstream control policy, or OPA. The recorded states are modeled scenario assertions used to make the boundary inspectable.

The control needs an explicit evidence map

The Control-to-Evidence Mapping Contract records which evidence requirements belong to a control before policy evaluation starts.

A minimal structure includes:

control_evidence_mapping:
  mapping_id:
  mapping_version:
  control_id:
  control_version:
  control_claim:
  evaluation_rule:
  evidence_requirements:
    - requirement_id:
      requirement_version:
      required:
      evidence_type:
      evidence_layer:
      minimum_objects:
      scope_ref:
      admissibility_required:

The useful part is the relationship between the control claim and its required proof inputs.

For each requirement, the assessment should be able to show what evidence type was expected, which scope it applied to, whether admissibility was required, and whether the requirement was actually satisfied.

Without that mapping, a pipeline can easily confuse:

everything received was valid

with:

everything required was received

Those statements are not equivalent.

Missing evidence should stay unresolved

Missing evidence is easy to mishandle because a binary control model wants a binary answer.

Consider an API timeout while collecting one required evidence type.

The timeout does not establish that the control condition failed. It establishes that the required observation was not obtained.

The state should remain explicit:

required evidence missing
-> evidence set incomplete
-> control evaluation blocked
-> control status unknown

This keeps collection failure separate from control failure.

The same applies when a source is unreachable, an expected object is absent, or the declared scope was only partially observed. Those conditions may require recollection or review, but they are not themselves proof that the target violated the control.

The mapping is part of replay history

The evidence map can change over time.

Suppose mapping version 0.1.0 requires certificate-state evidence only.

A later version adds runtime protocol-state evidence.

If an old assessment is replayed using the new mapping, it is being evaluated against a different proof requirement.

The historical path therefore includes more than evidence and policy:

control claim
-> evidence requirement map
-> admissible evidence set
-> policy input
-> policy evaluation
-> control result

Preserving the historical policy while discarding the historical evidence map still changes the question being replayed.

This is similar to policy versioning, but it occurs earlier in the pipeline. The evidence map determines what must exist before the policy is allowed to run.

Sufficiency still does not prove control correctness

A complete evidence set only proves completeness against the declared mapping.

The mapping itself may still be wrong.

It may omit an important evidence type. The declared scope may be too narrow. A requirement may point to an inappropriate source. A technically valid evidence object may not be the right proof for the claim being assessed.

The mapping therefore answers a limited question:

Did the declared evidence requirements get satisfied?

It does not establish that those were the correct requirements.

That remains a control-design and review problem.

Where this sits in the pipeline

With this addition, the current evidence path becomes:

source system
-> collector
-> raw observation
-> transformation
-> normalized fact
-> evidence admissibility
-> evidence sufficiency
-> policy input
-> policy evaluation
-> control result

Previous artifacts dealt with whether evidence survives collection, transformation and evaluation with enough provenance to be replayed.

This artifact adds a different check before policy execution: whether the declared proof set is actually complete.

A valid evidence object can still be insufficient for the claim being made.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM prescribes this mapping schema, the TLS example, an evidence-sufficiency gate, or OPA.

The engineering interpretation is narrower.

Before producing a control conclusion, the evidence package should preserve which proof inputs were required, which were present, which were admissible, and which remained missing.

Otherwise a valid piece of evidence can be used to support a conclusion that required more proof than the pipeline collected.

Public reference artifacts

The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline repository:

• Control-to-Evidence Mapping Contract

• Control-to-Evidence Mapping Synthetic Example

• Evidence Admissibility Gate

The current artifact models evidence sufficiency through a contract and synthetic examples.

It does not yet implement automatic evidence-set assembly, repo-wide evidence-requirement schema enforcement, control-catalog version governance, or cryptographic binding of a mapping artifact to an assessment result.

Origin and scope

CodeYourCompliance

Website: Compliance Evidence Automation | CodeYourCompliance

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• Invalid Evidence Is Not a Failed Control

• A Control Result Is Not Replayable Evidence

• From Evidence Principles to an Executable Control

The usual explanation is that something changed in the target system. That is possible, but it is not the only explanation. The evidence may be unchanged. The normalized facts may be unchanged. The evaluation context may also be unchanged.

The policy may have changed instead.

If the assessment history preserves only:

previous result: PASS
current result: FAIL

then the difference is visible, but its cause is not.

That becomes an evidence problem before it becomes a control problem.

The result depends on the policy that produced it

Policy-as-code makes evaluation repeatable. It does not make the result independent of the rule being evaluated.

The synthetic example in the public Policy Evaluation Provenance Example keeps one policy input constant:

days_to_expiry = 30
minimum_certificate_days_remaining = 30

Policy version 0.1.0 evaluates:

days_to_expiry >= minimum_certificate_days_remaining

The modeled result is PASS.

Policy version 0.2.0 evaluates:

days_to_expiry > minimum_certificate_days_remaining

The modeled result is FAIL.

The target-state identifier, normalized fact and evaluation context are held constant by construction. Only the policy semantics change.

The example does not prove that a real target remained unchanged between two assessments. It isolates one variable so that the decision difference can be attributed to the policy change inside the synthetic scenario.

The useful boundary is therefore narrower:

same policy input
+ different policy semantics
-> different decision

result difference alone
does not establish target-state change

A changed result needs more context before it can be called drift.

Policy version is only one part of evaluation provenance

Recording a field such as:

policy_version = 0.2.0

is useful, but replay needs more than a version label.

The Policy Evaluation Provenance Contract models the decision path with fields such as:

evaluation:
  engine:
  engine_version:
  policy_id:
  policy_version:
  rule_id:
  rule_version:
  policy_artifact_ref:
  policy_artifact_sha256:
  policy_input_ref:
  policy_input_sha256:
  evaluation_context_ref:
  evaluation_context_sha256:
  implementation_version:
  evaluated_at_utc:
  decision_executed:
  result:

The point is not to collect metadata for its own sake.

A reviewer needs to know which policy artifact was evaluated, which input it consumed, which context was applied, which decision entry point ran, and which engine and surrounding implementation produced the result.

Without that information, a historical assessment can easily be replayed using today’s policy instead of the policy that produced the original decision.

The evidence may be the same. The replay is not.

A policy hash does not establish policy authority

The policy artifact can also be hashed.

That improves identity. A SHA256 digest can show which policy bytes were referenced by an evaluation record and whether the preserved artifact later changed.

It does not show whether those bytes should have been used.

This distinction matters:

policy byte identity
!=
policy release authority

A digest does not prove that the policy was approved, current at the time of evaluation, applicable to the assessed scope, or released by an authorised control owner.

A replay package can therefore reproduce a decision against the wrong policy artifact with perfect integrity.

The replay is technically faithful. The policy selection may still be wrong.

Signed policy bundles, approval records and trusted policy-release provenance are separate concerns.

Deterministic evaluation does not validate the policy

OPA can evaluate the same input against the same Rego artifact and return the same result repeatedly.

That proves something useful: the identified evaluation path is deterministic.

It does not prove that the policy represents the correct control interpretation.

A threshold can be wrong. An exception may be missing. A rule may be valid for one system scope and unsuitable for another.

Evaluation provenance answers:

What policy path produced this result?

It does not answer:

Was this the correct policy for this control?

That judgment remains outside the policy engine.

Replay has to preserve the historical decision path

This repository already contains a bounded Replayable TLS Control.

Its pipeline is:

evidence schema
-> context schema
-> integrity
-> freshness
-> derived facts
-> policy evaluation
-> assessment result

That reference implementation records hashes for evidence, schemas, evaluation context and policy, together with implementation and runtime provenance.

The new policy-evaluation provenance artifact does not replace that implementation. It extracts one specific rule from it:

A historical control result should remain tied to the policy and evaluation path that produced it.

That makes this problem slightly different from the earlier question of whether a control result is replayable at all.

The first question is:

Can I reproduce this result?

The next question is:

Am I reproducing the historical decision,
or re-evaluating historical evidence
under a different policy?

Both operations may be useful.

They should not be confused.

The decision path has more than one moving part

Once collection, normalization and policy evaluation are separated, a changed assessment result can have several explanations:

target changed
collector changed
transformation changed
policy input changed
evaluation context changed
policy changed
engine or implementation changed

A PASS -> FAIL transition does not identify which one occurred.

The control result is the end of the pipeline. It is not a complete explanation of the pipeline.

Public reference artifacts

The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline:

• Policy Evaluation Provenance Contract

• Policy Evaluation Provenance Synthetic Example

• Replayable TLS Control

The synthetic comparison does not claim to be an OPA runtime capture. The repository also does not yet implement signed policy bundles, trusted policy-release attestation or generic cross-policy regression testing.

Those remain separate implementation steps.

A control result is downstream of evidence, context and policy.

If the policy path disappears, part of the decision history disappears with it.

Origin and scope

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• A Control Result Is Not Replayable Evidence

• Invalid Evidence Is Not a Failed Control

• From Evidence Principles to an Executable Control

The first ten notes focused mainly on vendor claims and the evidence behind them.

There is another evidence problem once the documentation looks complete.

A vendor may have a policy, workflow diagram, product documentation, and configuration showing how an AI agent is expected to behave.

That does not show whether the control is enforced when the agent actually runs.

Claim

The documentation says:

“High-risk actions require human approval.”

The policy defines the requirement.

The workflow contains an approval step.

The product may provide a setting for it.

These are useful evidence of the intended control.

They do not show that every qualifying action is actually stopped and sent for approval.

Why it sounds sufficient

Policies and configuration are familiar review artifacts.

If the approval rule is documented and the feature is enabled, it is easy to record human oversight as an implemented control.

The problem is the execution path.

An agent may select tools, call APIs, retry failed actions, or reach the same result through a different sequence.

A reviewer therefore still needs to know:

• what triggers approval;

• whether execution stops before the action;

• what happens after rejection;

• how retries are handled;

• whether another tool path can avoid the trigger;

• what happens if the approval mechanism fails;

• whether the approval can be linked to the final action.

Those questions cannot be answered by the policy alone.

What it actually proves

A policy can show that the control was defined.

Product documentation can show that an approval mechanism exists.

Configuration evidence can show that it was enabled.

These support control design and configuration.

Evidence of enforcement comes from somewhere else.

For example, an execution record could show:

Trigger: The agent attempted an action requiring review.

Pause: The action stopped before execution.

Reviewer: The request was sent to an attributable reviewer.

Decision: Approval or rejection was recorded.

Execution: The action occurred only after approval.

Trace: The approval event and final action can be linked.

That is evidence about what happened, not only what was supposed to happen.

A public evidence example

The CodeYourCompliance Trust Signal profile for Decagon shows the distinction at the public-evidence layer.

The profile identifies security and AI data-usage surfaces, along with enterprise evidence such as audit logs and AI guardrails. It also keeps an explicit boundary: those public security materials are review surfaces, not implementation verification.

That boundary becomes important for agent controls.

Trust Signal can help locate the documented evidence surface.

It cannot show, by itself, whether a guardrail or approval control was enforced in a particular execution.

What it does not prove

A documented approval control does not automatically establish:

Trigger coverage. The rule may exist without showing that every relevant action is detected.

Bypass resistance. Another tool or workflow may reach the same outcome without triggering approval.

Failure behavior. Documentation may not explain what happens when the approval service fails or times out.

Retry behavior. A rejected action may be attempted again through a different path.

Sequence. An approval record is weak evidence if the action could occur before approval was captured.

Version applicability. Evidence collected against one model, prompt, tool configuration, or agent version may not remain valid after those components change.

Weak-answer pattern

A common answer is:

“Human approval is required for high-risk actions. See our policy and product documentation.”

That may be enough to support the existence and design of the control.

The gap appears when the review stops there.

The buyer has evidence of the rule, but not yet evidence that the rule intercepted the relevant action during execution.

Evidence request

For controls that limit what an agent can do, ask for evidence from the execution path:

Provide evidence showing how actions requiring human approval are identified, paused, presented for review, approved or rejected, and linked to the final execution record. Include rejected actions, retries, alternate paths, and approval-service failure where available.

Useful evidence may include execution traces, approval logs, reviewer identity or role, timestamps, tool-call records, rejected-action records, failure tests, and bypass tests.

The purpose is not to collect more logs.

It is to connect the documented control to the behavior it is meant to constrain.

Review note

Human approval is documented as a control for high-risk actions. The available evidence supports control design, but enforcement across triggering conditions, retries, alternate execution paths, and failure conditions has not yet been demonstrated.

Usage boundary

Until enforcement evidence is available, a documented approval requirement should not be treated as proof that high-impact autonomous actions are consistently intercepted.

For higher-impact use cases, keep the agent within a boundary where failure or bypass of the approval step cannot independently create a consequential action.

The evidence question is no longer only:

What does the policy say?

It is also:

What happened when the control was tested?

Boundary

This material is for evidence structuring and review preparation.

The goal is to make the gap between documented controls and demonstrated behavior visible before those controls are relied upon.

#AIAssurance #AIGovernance #AIAgentTesting

status = not_configured

A normalization layer turns that into:

enabled = false

The second value is easier for a policy engine to consume.

It is also not what the system originally returned.

Someone decided that not_configured should mean false.

That decision happened before policy evaluation.

If the evidence package keeps only the normalized value, the interpretation disappears while its consequence survives.

That is the problem.

Raw observation and normalized fact are different evidence layers

Most systems do not produce evidence in the format a control policy wants.

Commands return text. APIs use vendor-specific fields. Configuration files use their own syntax. One source may say enabled, another 1, another active.

Normalization is necessary.

A typical path looks like this:

raw evidence
→ normalized fact
→ derived fact
→ policy input
→ policy result

The public CodeYourCompliance Evidence Contract already separates these layers.

The distinction is not about data formatting.

It is about where interpretation enters the evidence chain.

The raw observation records what the source returned.

The normalized fact records how the pipeline chose to represent it.

Those are related objects.

They are not the same object.

The same observation can produce different facts

The new public Transformation Provenance Example isolates this problem.

The synthetic raw evidence is held constant:

status = not_configured

Normalization version 0.1.0 applies this rule:

MAP_NOT_CONFIGURED_TO_FALSE

and produces:

enabled = false
state = resolved

Version 0.2.0 uses a different rule:

PRESERVE_NOT_CONFIGURED_AS_UNKNOWN

and produces:

enabled = null
state = unknown

Same raw observation.

Different transformation logic.

Different normalized fact.

The synthetic example deliberately holds the raw observation and target-state identifier constant so that only the transformation changes.

It does not establish which normalization rule is correct.

It establishes a narrower point:

same raw observation
+ transformation difference
→ normalized fact changed
→ target-state change is not established
  by this difference alone

That boundary matters whenever normalized facts are later compared for drift or fed into policy evaluation.

Transformation provenance should travel with the fact

The public Transformation Provenance Contract records the minimum context needed to inspect that path.

The current pattern includes fields such as:

transformation:
  name:
  version:
  rule_id:
  rule_version:
  input_ref:
  input_field:
  input_value:
  output_field:
  output_value:
  output_state:
  executed_at_utc:

This means a reviewer does not only see:

enabled = false

They can also determine which raw observation was consumed, which rule performed the mapping, which implementation version ran, and what output it produced.

That is provenance.

It is not attestation.

Transformation provenance is not transformation correctness

A versioned rule can still be wrong.

A documented mapping can still be unsuitable for a particular control.

A normalizer can contain a bug.

Recording:

rule_version = 0.2.0

does not prove that version 0.2.0 interpreted the source correctly.

It only makes that interpretation identifiable.

This is the same boundary discussed in The Collector Is Part of the Evidence: preserving how something entered the evidence chain is different from proving that the mechanism was correct.

The collector can be wrong.

The transformation can be wrong.

Provenance gives the reviewer something concrete to challenge.

Integrity does not solve the semantic problem

Hashing solves another problem.

Suppose the pipeline records:

source value:
not_configured

normalization:
not_configured → false

integrity:
verified

The integrity check can show that the normalized evidence object has not changed after sealing.

It cannot tell us whether false was the right interpretation in the first place.

A perfectly intact evidence package can preserve a bad transformation.

So these two claims need to remain separate:

integrity verification
!=
semantic correctness

Hashing protects the preserved object.

Transformation provenance explains how that object acquired its meaning.

Policy evaluation starts later than it looks

OPA may receive a clean object such as:

{
  "enabled": false
}

From OPA’s perspective, that input is simple.

From an evidence perspective, it already contains history.

A useful trace is closer to:

control result
→ policy input
→ normalized fact
→ transformation rule
→ raw observation
→ collector
→ source system

This also connects to the admissibility boundary described in Invalid Evidence Is Not a Failed Control.

The policy engine is not where the evidence story begins.

By the time evaluation starts, collection and interpretation have already happened.

If those steps disappear, the final result may still be deterministic while the path that produced its input is not inspectable.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM requires transformation metadata, rule versions, normalization schemas, OPA, or this implementation pattern.

The engineering question is narrower:

Can a reviewer determine how a raw system observation became the fact that was actually evaluated?

If not, the pipeline preserves the control result more carefully than the interpretation that created its input.

Public reference artifacts

The transformation provenance boundary is implemented as a public engineering artifact in the codeyourcompliance/evidence-validation-pipeline repository.

Main references:

• Transformation Provenance Contract

• Transformation Provenance Synthetic Example

• Evidence Contract

The repository currently models transformation provenance through structured metadata and synthetic examples.

It does not yet cryptographically bind transformation or rule digests to normalized facts. It also does not implement automated transformation-version equivalence or regression testing, or repo-wide transformation provenance schema enforcement.

Those are separate implementation steps.

Origin and scope

CodeYourCompliance

Website:

Compliance Evidence Automation | CodeYourCompliance

CodeYourCompliance explores compliance automation, replayable audit evidence, read-only evidence collection, policy-as-code, and evidence packages.

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• The Collector Is Part of the Evidence

• Invalid Evidence Is Not a Failed Control

• Compliance Automation Starts at Evidence

The first nine notes looked at different AI vendor claims: training use, Trust Centers, SOC 2 scope, human review, subprocessors, retention, audit logs, data residency, and questionnaire answers.

The subjects were different.

The review problem was usually the same.

A vendor statement was available, but the supporting evidence was narrower than the statement suggested.

A claim-evidence matrix keeps that difference visible.

Claim

The review file says:

“Vendor assessment completed.”

The questionnaire has been returned.

The Trust Center, DPA, SOC 2 material, subprocessor information, and product documentation have been reviewed.

That shows that a review took place.

It does not show how individual claims were supported.

The file may still contain statements such as:

Customer data is not used for training.

Human review is limited.

Data residency is supported.

Audit logs are available.

Each statement may have a supporting source.

The question is what that source actually covers.

Why it sounds sufficient

Procurement reviews already collect a lot of material.

Adding another matrix can look like duplicate work.

The problem appears when answers and evidence are stored separately.

The questionnaire contains the claim.

The Trust Center contains one source.

The DPA may contain a stronger commitment.

Product documentation may narrow the feature scope.

A SOC 2 report may cover only part of the relevant environment.

If those relationships are not recorded, another reviewer has to reconstruct the reasoning later.

What it actually proves

A claim-evidence matrix records that relationship.

For each material claim, capture:

Claim: What did the vendor state?

Evidence source: What document or page supports it?

Scope: Which product, plan, feature, region, and data type are covered?

Exceptions: What sits outside the statement?

Evidence date: When was the source checked?

Evidence gap: What remains unresolved?

Buyer question: What needs to be clarified next?

Usage boundary: What should remain limited while the gap is open?

The useful sequence is:

vendor claim → evidence source → scope → evidence gap → buyer question → usage boundary

A simple example

Take:

“We do not train on customer data.”

The review should not stop at “No training.”

The matrix may show:

Source: Enterprise data-use documentation.

Scope: The statement applies to a defined commercial product.

Exceptions: Support, abuse monitoring, human review, or opt-in workflows require separate checking.

Evidence gap: Retention, logging, subprocessor handling, and deletion are not established by the training statement.

Buyer question: Which sources establish those remaining data-handling boundaries?

The vendor answer has not changed.

The assumptions around it are now visible.

A public evidence example

The CodeYourCompliance public evidence profile for Cohere separates public evidence surfaces rather than treating the vendor name as one source.

That is useful for locating the source side of the review.

But the profile does not decide whether a particular claim is sufficient for a buyer’s product, plan, use case, data type, region, or contract.

The claim-evidence matrix performs that next step:

evidence surface → claim → scope → remaining gap

Trust Signal helps locate and structure public evidence.

The matrix records what that evidence can actually support in the review.

What it does not prove

The matrix does not prove that a vendor is low risk.

It does not make a source stronger than its scope.

A SOC 2 report can still leave the AI data path unclear.

A subprocessor list can identify third parties without showing actual routing.

A Trust Center can provide useful evidence without resolving the buyer’s use case.

The matrix records those limits.

It does not remove them.

Weak-answer pattern

A common review pattern is:

Claim identified. Supporting document attached. Item closed.

The document may be relevant, but the review file does not record:

• which part of the claim it supports;

• which product or plan is covered;

• whether exceptions exist;

• whether another source is required;

• when the evidence was checked.

The result is a documentation package without a clear evidence chain.

Evidence request

For material claims, ask for the source and its boundary.

For each material AI data-handling, security, privacy, logging, retention, subprocessor, residency, and human-access claim, identify the supporting source, applicable product and plan, covered data types, known exceptions, evidence date, and contractual basis where applicable.

The goal is not more documents.

It is to identify which claims still lack usable evidence.

Review note

Material vendor claims have been mapped to available supporting sources. Some remain partially evidenced because product scope, data-type coverage, exceptions, evidence freshness, or contractual applicability are incomplete.

That is more useful than a spreadsheet containing only Yes, No, and See Trust Center.

Usage boundary

The matrix should always be read in context.

The same claim may support different use depending on the product, plan, data type, region, contract terms, and evidence date.

If the evidence supports only low-sensitivity internal use, record that.

If customer data requires stronger retention or subprocessor evidence, keep that limitation open.

A claim-evidence matrix is not a vendor score.

It is a dated record of what the available evidence supports and what remains unresolved.

The first nine notes focused on vendor statements and supporting sources.

The same question continues after documentation:

What evidence shows that a documented control actually works as described?

Boundary

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample claim-evidence matrix.

#AIVendorRiskAssessment #ThirdPartyRisk #AIGovernance

Six months later, someone reviews the assessment and asks how that expiry date was collected.

Was it read directly from the certificate file? Parsed from command output? Returned by an application API?

If the evidence package does not say, the value may still be intact, but part of its history is missing.

The collector is not just transport between a system and an evidence file. It is part of the chain that produced the observation.

The same field can have different origins

Take a field such as:

certificate_not_after

One collector might read a certificate file directly. Another might call openssl and parse stdout. Another might query an application endpoint.

The final value could be identical in all three cases.

That does not make the collection paths equivalent.

Each method depends on a different source and different interpretation logic. If only the normalized field is retained, those differences disappear.

This becomes more important when results change.

If yesterday’s collector reported one value and today’s collector reports another, the first question is not necessarily whether the target changed.

The collection path may have changed instead.

Record the collector with the observation

The current CodeYourCompliance evidence pattern records collector metadata with the observation:

{
  "collector": {
    "name": "ansible-readonly-tls-collector",
    "version": "0.1.0",
    "mode": "read_only",
    "method": "remote_command_and_file_read",
    "changed_target_system": false
  }
}

The collector name identifies what produced the observation.

The version identifies which implementation was running.

The method records how the state was obtained.

The read-only fields record the intended collection boundary and whether collection changed the target.

The public Collector Provenance Contract extends this with fields such as parser version and source reference.

None of these fields proves that the collector was correct.

They make the production path inspectable.

Read-only does not solve provenance

I have previously treated read-only collection as an audit boundary.

That boundary still matters.

A collector that repairs a system while collecting evidence mixes observation with remediation. Once that happens, it becomes harder to tell whether the recorded state existed before the collector touched the target.

But a collector can be read-only and still produce misleading evidence.

A parser can be wrong.

A configuration path can change.

A missing value can become false when it should remain unknown.

None of those problems requires a write operation.

Two questions therefore remain separate:

The first is about collection restraint.

The second is about provenance.

Collector changes can look like system changes

The public collector provenance example makes this problem explicit.

It contains two synthetic evidence objects for the same service and the same synthetic target-state identifier.

The first uses collector version 0.1.0, reads the configured certificate file and records:

not_after = 2026-09-30T00:00:00Z

The second uses collector version 0.2.0, performs a runtime TLS handshake and records:

not_after = 2026-10-15T00:00:00Z

The target-state identifier is deliberately held constant by the scenario.

The collector version, collection method, parser version and source reference change.

The example does not establish which observation is correct.

It establishes a narrower boundary:

Before a difference is classified as system drift, the collection path needs to remain visible.

This is also why retaining only the final PASS or FAIL is weak. As discussed in Invalid Evidence Is Not a Failed Control, a result only has meaning if the pipeline preserves how that result was reached.

Normalization has the same problem

Raw collection is often only the first step.

A command returns text. A parser extracts values. A normalization layer converts them into stable fields. A policy engine evaluates those fields.

The useful trace is closer to:

Not every implementation detail belongs in the final report.

But the evidence package should retain enough metadata to identify which collection and transformation path produced the evaluated fact.

Otherwise the policy side is visible while the collection side remains opaque.

Provenance is not proof of correctness

There is another boundary worth keeping clear.

Recording collector identity, version and method does not make the collector trustworthy.

A known collector can contain a bug.

A read_only attribute can be wrong.

A documented source can still be unsuitable for the control being evaluated.

Collector metadata provides a trail to inspect.

It does not validate that trail.

Structured provenance improves traceability. It does not turn implementation claims into facts.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM requires Ansible, collector version fields, parser metadata or this evidence model.

The engineering interpretation is narrower.

If system state is being turned into evidence for later control assessment, the collection path should not disappear once the normalized value has been produced.

A reviewer should be able to determine what collected the observation, how it was collected, what source was used and which implementation version was involved.

If that information is missing, part of the evidence history is missing too.

Public reference artifacts

The collector provenance boundary described here is implemented as a public engineering artifact in the codeyourcompliance/evidence-validation-pipeline repository.

Main references:

• Collector Provenance Contract

• Collector Provenance Synthetic Example

• Minimal Evidence Object

The contract defines the provenance fields needed to keep collector identity, version, method, parser and source attached to an observation.

The synthetic example shows why an observation difference should not automatically be classified as target-state drift when the collector also changed.

The repository does not yet cryptographically bind collector implementation digests to evidence objects. It also does not implement automated collector-version equivalence testing.

Those remain separate implementation steps.

Origin and scope

CodeYourCompliance

Website:

Compliance Evidence Automation | CodeYourCompliance

CodeYourCompliance explores compliance automation, replayable audit evidence, read-only evidence collection, policy-as-code, and evidence packages.

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• Read-Only Collection as an Audit Boundary

• Compliance Automation Starts at Evidence

• Invalid Evidence Is Not a Failed Control

A completed questionnaire can still leave the buyer without usable evidence.

AI vendor reviews often ask:

“Do you use customer data for model training?”

“Do you support data residency?”

“Are audit logs available?”

“Do you maintain a subprocessor list?”

The vendor answers.

The questionnaire is marked complete.

The review may still be incomplete.

The question is not only:

Did the vendor answer?

The better question is:

What source supports the answer, what does that source cover, and what remains unresolved?

A questionnaire collects statements.

An evidence review tests whether those statements support the buyer’s intended use.

Claim

The review file says:

“Vendor questionnaire completed.”

Or:

“Vendor confirmed required controls are in place.”

That establishes that the vendor responded.

It does not establish that the answers are evidence-complete.

Why it sounds sufficient

Questionnaires create structure.

They force buyers to ask about security, privacy, retention, subprocessors, model training, audit logs, data residency, and access controls.

That is useful.

The problem starts when the answer becomes the evidence.

A questionnaire may contain:

No customer data is used for training.

SOC 2 Type II available.

Audit logs supported.

Data stored in the United States.

See Trust Center.

Each answer may be accurate.

But none of them tells the buyer, by itself:

• which source supports it;

• which product and plan it covers;

• which data types are included;

• which exceptions apply;

• whether the evidence is current;

• whether the commitment is contractual.

The questionnaire records what the vendor said.

It does not automatically record what the buyer can rely on.

What it actually proves

A completed questionnaire may establish that the vendor responded to a defined set of questions and that relevant claims have been collected.

It can also identify areas that need follow-up.

It does not establish the strength or scope of the evidence behind each answer.

Consider:

Question: Do you use customer data for model training?

Vendor answer: No.

The review file still needs:

Source: Where is that commitment written?

Scope: Which product, plan, endpoint, or feature does it apply to?

Data covered: Prompts, outputs, files, metadata, logs, or all customer content?

Exceptions: Support, abuse monitoring, opt-in evaluation, or other workflows?

Evidence date: When was the source checked?

Contract boundary: Is this public documentation or part of the customer agreement?

Gap: What remains unresolved?

That is the difference between an answer and an evidence record.

A public evidence example

The CodeYourCompliance public evidence profile for Cohere shows how a vendor review can move beyond a questionnaire answer.

The profile separates public evidence surfaces rather than treating the company name as one source.

A response such as:

“SOC 2 available.”

or:

“Customer data is not used for training.”

still needs to be mapped to the specific source that supports it.

The reviewer then needs to check the product and plan scope, evidence date, data types covered, known exceptions, and whether a stronger contractual or customer-specific source is required.

The profile helps locate evidence surfaces.

It does not make the review conclusion.

That distinction matters.

Questionnaire answer → evidence source → scope → evidence gap

That is evidence conversion.

What it does not prove

A completed questionnaire does not automatically establish:

Source strength. “Yes,” “No,” or “See Trust Center” does not show whether the source is contractual, audited, product documentation, marketing material, or unsupported.

Product scope. A company-level answer may not apply to the exact product, plan, feature, endpoint, or integration.

Data-path scope. A training answer does not establish retention. A residency answer does not establish processing location. A subprocessor answer does not establish routing.

Freshness and exceptions. Supporting documents may change, and binary answers may hide support access, safety review, fallback providers, beta features, or configurable behaviour.

Contract applicability. Public documentation may not create the same commitment as a DPA, order form, or product addendum.

None of these gaps means the vendor gave a false answer.

They mean the answer still needs evidence mapping.

Weak-answer pattern

This is the answer-without-source pattern.

The buyer asks:

“Are audit logs available?”

The vendor answers:

“Yes.”

The spreadsheet becomes green.

But the review file does not identify which events are logged, which plan is covered, how long logs are retained, whether customers can export them, or whether AI-specific activity is visible.

The questionnaire looks complete.

The evidence record is not.

Evidence request

Do not ask the vendor to complete another questionnaire.

Ask the vendor to support the existing answers.

For each material AI, security, privacy, retention, logging, subprocessor, data-residency, and human-access claim, identify the supporting source, applicable product and plan, relevant data types, known exceptions, evidence date, and contractual basis where applicable.

For higher-impact claims, ask:

• Does the source apply to the exact product and feature?

• Which data types are explicitly covered?

• Which exceptions exist?

• Has the supporting evidence changed?

• Is stronger private or customer-specific evidence required?

The goal is not more questions.

The goal is claim-to-evidence mapping.

Review note

The vendor completed the requested questionnaire, but several material responses have not yet been mapped to product-specific supporting evidence. Source strength, product and plan scope, data-type coverage, exceptions, evidence freshness, and contractual applicability remain incomplete for key AI data-handling claims.

The questionnaire is complete.

The evidence mapping is not.

Usage boundary

Until material answers are mapped to evidence:

Limit use to low-sensitivity internal or test data. Do not expand to customer-confidential data, regulated data, source code, employee records, autonomous actions, or externally relied-upon outputs where unsupported questionnaire answers would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit remains:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A questionnaire can collect claims.

It cannot determine whether those claims are supported.

The useful sequence is:

vendor answer → evidence source → scope → evidence gap → buyer question → usage boundary

A completed questionnaire is an input.

It is not an evidence-complete review.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRiskAssessment #ThirdPartyRisk #AIGovernance

The collection partially fails.

The output file still exists.

A downstream policy engine reads the file, cannot find the expected field, and returns fail.

The report now says the control failed.

But the evidence may have failed instead.

That distinction has to survive the pipeline.

If it does not, compliance automation can manufacture control failures from collection failures, integrity failures, or invalid evaluation context.

The audit problem starts before the policy engine.

A file is not automatically admissible evidence

A common pipeline looks like this:

collect
  ↓
evaluate
  ↓
pass / fail
  ↓
report

The hidden assumption is that collection produced usable evidence.

That assumption is unsafe.

An artifact can exist while still being unusable. Its schema may be invalid, its integrity check may fail, its freshness window may have expired, or the evaluation context may be malformed.

None of those conditions proves that the control failed.

They mean the system lacks a defensible basis for a control decision.

That requires an explicit gate.

Evidence admissibility is not control evaluation

The pipeline should answer two different questions.

First:

Is this evidence admissible for evaluation?

Then:

Does the observed condition satisfy the control policy?

Those questions should not share the same result field.

The CodeYourCompliance reference implementation uses this sequence:

evidence schema gate
→ context schema gate
→ integrity gate
→ freshness gate
→ derived facts
→ decision evaluation
→ assessment result

If an earlier gate fails, the control decision does not execute.

The state remains:

decision_executed = false
control_status = unknown

That is not a failed control.

It is a blocked decision.

Invalid evidence should stop the pipeline

Take an integrity mismatch.

The evidence package no longer matches the sealed object the pipeline expected.

The wrong transition is:

invalid_evidence
→ control_status = fail

The correct transition is:

invalid_evidence
→ decision_executed = false
→ control_status = unknown

The policy engine has nothing admissible to evaluate.

Returning fail would create a control conclusion that the evidence does not support.

The implementation rule is simple:

No admissible evidence.
No control decision.

Stale evidence uses the same boundary

Freshness is a different failure mode, but the same gate applies.

Evidence may still have a valid integrity hash and known provenance while being too old for the current evaluation window.

That evidence is not corrupt.

It is stale.

A freshness policy may classify it as stale_evidence, but that classification belongs to evidence admissibility.

It is not the downstream control decision.

This is why the public example separates:

admissibility_evaluation

from:

control_decision

The control decision remains unexecuted and unknown.

Evidence evaluation is not control evaluation.

OPA is not the boundary

OPA can evaluate either kind of policy.

A Rego freshness policy can decide whether evidence may proceed. A control policy decides whether admissible facts satisfy a declared condition.

The tool is not the point.

The stage is.

Those decisions should remain separate even if both are implemented in OPA.

The state transition should be explicit

The useful rule is not a longer list of statuses.

It is a prohibited transition.

This should never happen:

invalid_evidence
→ control_status = fail

Neither should this:

stale_evidence
→ control_status = fail

The allowed path is narrower:

admissible evidence
→ decision executed
→ pass / fail

Everything before that belongs to evidence qualification.

fail means the control policy evaluated admissible evidence and found the condition unsatisfied.

unknown means the pipeline could not reach that judgment.

Reports should preserve the blocked decision

A reporting layer should not translate every non-pass state into non-compliant.

It should preserve the failed gate, evidence status, whether the decision executed, and the control status.

Reports summarize a decision path.

They should not invent one.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM prescribes this state machine, OPA, Rego, SHA256, or this implementation.

The engineering question is narrower:

Can the evidence pipeline distinguish failure of the control from failure of the material used to assess it?

If not, the final report may look deterministic while its underlying decision path is not.

A failed control is a control conclusion.

Invalid evidence is the absence of a defensible basis for that conclusion.

That is the boundary.

Public reference artifacts

The admissibility boundary described here is implemented in the public codeyourcompliance/evidence-validation-pipeline repository.

The main reference package is:

• examples/replayable-tls-control

• docs/evidence-admissibility-gate.md

The TLS example separates evidence validation from control decision evaluation. Schema validation, integrity verification, freshness checks and evaluation-context validation occur before the control decision.

If a gate fails, the decision does not execute. The result remains unknown, not fail.

These examples do not prove that the underlying system is compliant. They show whether the pipeline had a defensible basis to make a control decision at all.

They also stop before remediation. Certificate renewal, configuration changes and other corrective actions belong to a separate workflow with separate permissions and evidence.

Origin and scope

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• From Evidence Principles to an Executable Control

• Can Your Audit Evidence Survive Replay?

• Why Audit Evidence Goes Stale

The remediation owner says the work is complete.

There is evidence.

But does that evidence actually support the closure requirements?

That is the problem in the first Awesome GRC Workflows case.

Evidence attached ≠ requirement satisfied

A document existing in the package tells you very little by itself.

Take a test result.

The useful questions are not:

Is there a test document?

They are:

What requirement is this supposed to support?

What was actually tested?

When was it tested?

What does the result really establish?

The same applies to other artifacts.

An evidence inventory may say a procedure exists.

That does not mean you have reviewed the procedure.

A management statement may provide context.

That does not turn it into independent verification.

A deployment record may show that a change happened.

That does not automatically prove sustained effectiveness.

The reviewer has to connect:

requirement
→ evidence
→ limitation
→ conclusion

That is the work.

The case

In Remediation Evidence Closure Review, you act as an Evidence Review Analyst.

You receive a fictional remediation closure package containing documented requirements and multiple forms of synthetic evidence.

Your task is to review the submitted package and determine what the evidence can actually support.

You produce three deliverables:

1. Evidence-to-requirement review workpaper

2. Evidence deficiency register

3. Closure-readiness memo

This is not a quiz.

You have to inspect the records, identify gaps and contradictions, document what remains unresolved, and leave behind work another reviewer can follow.

Recommendation ≠ approval

There is another boundary in the exercise.

You may analyze the evidence.

You may identify deficiencies.

You may request additional evidence.

You may prepare a closure-readiness recommendation.

You may not close the issue.

You may not accept residual risk.

You may not make legal determinations or execute remediation.

Correct analysis does not create approval authority.

That distinction is part of the work sample.

Run the case

The case is synthetic. The organization, records, systems, people, and evidence are fictional or safely constructed.

Start with the README, scenario, task brief, inputs, and templates.

Complete the three deliverables before opening the reviewer guide or reference answer.

Your job is not to guess what happened.

Your job is to determine what the submitted evidence can actually support.

Run the case.

https://github.com/codeyourcompliance/awesome-grc-workflows/tree/main/cases/remediation-evidence-closure-review

A country name sounds like a boundary.

“Customer data is stored in the United States.”

“EU data residency is available.”

“Enterprise customers can select a region.”

These statements feel specific.

They identify a place, and a place can make the data path appear settled.

But in an AI vendor review, location is only the beginning of the question.

A residency statement may describe where one category of data is stored while saying little about where the rest of the workflow is processed, cached, backed up, reviewed, or routed.

The review question is not only:

Where is customer data stored?

The better question is:

Which data is covered, what happens to it, which systems and providers receive it, and which parts of that path are actually restricted to the stated region?

A hosting location can be accurate while the geographic boundary remains incomplete.

That is the evidence gap.

Claim

The vendor says:

“Customer data is stored in the United States.”

Or:

“EU data residency is available.”

Or:

“Enterprise customers can select a hosting region.”

These statements may establish that defined customer data is stored within a stated location.

They do not establish the complete geographic path of the buyer’s data.

Why it sounds sufficient

Data residency is often reduced to a country, region, or dropdown selection.

That works well in a questionnaire.

The buyer asks where customer data is hosted. The vendor names a location. The answer appears complete.

But an AI workflow rarely involves one database and one data object.

A single interaction may create:

• prompts and outputs;

• uploaded files;

• audio and transcripts;

• embeddings and indexes;

• metadata and usage records;

• logs and moderation events;

• support artifacts;

• cached copies;

• backups and recovery copies.

The application may store customer records in one region while a model provider, transcription service, analytics platform, support tool, or security service operates through a different path.

The storage location may be clear.

The processing location may not be.

The primary database may sit in one country.

Backups, logs, support systems, local caches, or downstream providers may follow different rules.

This is how a true statement can create the wrong impression.

“Stored in the United States” may accurately describe the primary application environment.

It may not describe the entire service.

What it actually proves

A current, product-specific residency statement may establish that:

• a primary hosting region has been identified;

• defined customer content is stored in a stated location;

• a regional storage option exists;

• the vendor has documented a geographic control;

• a particular plan or deployment supports regional selection.

These are meaningful evidence points.

They establish something about location.

They do not automatically establish:

• which data types are covered;

• whether storage and processing occur in the same region;

• where external providers operate;

• where backups and logs remain;

• where support personnel may access data;

• whether exceptions alter the path;

• whether the commitment is contractual;

• whether the same boundary applies to every product, plan, and feature.

Storage is part of the data path.

It is not the whole data path.

A public evidence example

The CodeYourCompliance public evidence profile for Granola records a US data-residency reference among the public evidence surfaces observed for Granola’s AI meeting-notes product.

The profile was last checked on 27 June 2026. It also records that regional residency outside the United States was not observed during that review.

The profile is an evidence index.

It is not implementation verification, a security assessment, or a vendor recommendation.

Granola’s Security, Privacy & Data FAQs states that data is stored on AWS servers in the United States and that EU, UK, Canadian, Australian, and other regional residency options are not currently offered.

The same documentation also says that meeting notes are cached locally on the user’s device.

Granola’s separate security page states that notes are stored in a US-hosted AWS Virtual Private Cloud and backed up daily. It also explains that transcription providers and AI providers form part of the service workflow.

These statements are useful.

They identify the primary hosting environment.

They identify the absence of alternative regional residency options.

They also begin to show that the service involves more than one data location and more than one provider role.

They do not, by themselves, establish:

• where each transcription or AI provider processes data;

• which data types each provider receives;

• where backups and recovery copies are located;

• whether local caches follow the same deletion lifecycle;

• where support or incident-response access may occur;

• whether every plan and feature follows the same path;

• which geographic commitments appear in the customer’s contract.

The public profile helps locate the evidence.

The vendor’s own documentation remains the primary source.

Neither should be stretched beyond what it says.

What it does not prove

A data-residency statement does not automatically establish:

Data-type coverage. The statement may apply to notes, prompts, or outputs without covering uploaded files, transcripts, embeddings, metadata, logs, moderation records, support copies, or backups.

Processing location. Data can be stored in one country while being processed elsewhere by model, transcription, security, analytics, or infrastructure providers.

Secondary storage. Caches, indexes, vector stores, observability platforms, disaster-recovery environments, and support tools may follow different location rules.

Provider routing. The buyer may not know which model or service provider receives a request, whether fallback routing occurs, or whether provider selection changes by feature or region.

Human access location. A regional storage claim does not show where support, engineering, safety, or incident-response personnel may access customer data.

Exception paths. Abuse review, legal requests, security investigations, support cases, customer-enabled integrations, and service failures may alter the normal path.

Deletion scope. Deletion from the main application does not automatically establish deletion from local caches, logs, support systems, backups, or downstream providers.

Product and plan scope. A location statement may apply to one application, feature, plan, region, or deployment model without covering the exact service under review.

Contract scope. A public documentation page does not necessarily provide the same commitment as a DPA, order form, service agreement, regional addendum, or negotiated term.

None of these gaps proves that the vendor is handling data improperly.

They show that the location claim is narrower than the decision the buyer is trying to make.

Weak-answer pattern

This is the country-without-path pattern.

The vendor names a hosting region.

The buyer records:

“Data hosted in the United States.”

Or:

“EU residency available.”

But the review file does not identify:

• which data types are covered;

• where processing occurs;

• which providers receive the data;

• where logs and backups remain;

• where support access may occur;

• which exceptions change the path;

• which product and plan are covered;

• which contract term supports the boundary.

The answer may be accurate.

The geographic scope is still incomplete.

Evidence request

Do not ask only:

“Where is customer data hosted?”

Ask for the data-location map.

A stronger request is:

Please provide a product- and plan-specific data-location map for the intended workflow. For each relevant data type, identify where it is stored, where it is processed, which providers receive it, where backups and recovery copies are held, where human or support access may occur, which exceptions may alter the path, and which contractual source supports the stated regional boundary.

Then ask:

• Which data types are covered by the residency commitment?

• Does the stated region apply to both storage and processing?

• Which model, transcription, infrastructure, security, and analytics providers receive data?

• Can routing or fallback move processing to another provider or region?

• Where are logs, caches, indexes, backups, and recovery copies located?

• From which locations may support, engineering, safety, or incident personnel access the data?

• Do integrations or customer-enabled features change the path?

• Does the boundary differ by product, plan, configuration, or feature?

• Which DPA, order-form term, or regional addendum confirms the commitment?

The vendor does not need to disclose sensitive architecture.

It does need to establish a geographic boundary that the buyer can use.

Review note

A weak review note says:

“Customer data is hosted in the United States.”

A stronger review note says:

The vendor identifies a customer-data hosting region, but the available evidence does not establish the complete geographic scope of processing, provider routing, logs, caches, backups, support access, subprocessors, or exception paths. The response is not evidence-complete for a use case requiring verified in-region storage and processing.

That note does not say the residency statement is false.

It prevents a storage-location claim from being treated as proof of the complete data path.

Usage boundary

Until the geographic path is evidenced:

Limit use to data that does not require verified in-region storage and processing. Do not introduce regulated, highly confidential, customer-restricted, or contract-restricted data where uncertainty about processing locations, providers, backups, support access, or exception paths would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A residency claim can support that review.

It cannot replace the data path.

A country name is evidence.

It is not regional coverage.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRiskAssessment #DataResidency #AIGovernance

AI vendors often say:

“Audit logs are available.”

“Enterprise administrators can monitor activity.”

“Logs can be exported to your SIEM.”

These statements sound like evidence of accountability.

They may prove that some logging capability exists.

They do not prove that the events needed to reconstruct an AI workflow are actually recorded.

The review question is not only:

Does the vendor provide audit logs?

The better question is:

Which events are recorded, which events are missing, and can the buyer reconstruct what the AI system accessed, produced, changed, or exposed?

An audit log can exist while the decisive AI event remains invisible.

That is the evidence gap.

Claim

The vendor says:

“Enterprise customers have access to audit logs.”

Or:

“Key platform activity is logged.”

Or:

“Security and administrative events can be exported.”

These statements may establish that the product records certain identity, administrative, operational, or security events.

They do not establish that the actual AI activity is visible.

Why it sounds sufficient

Audit logs are familiar enterprise controls.

They suggest that administrators can monitor activity, investigate incidents, identify configuration changes, and retain evidence for internal review.

A vendor questionnaire may therefore ask whether audit logs exist and treat a “yes” answer as complete.

But conventional SaaS logs often focus on:

• login and logout;

• account creation;

• role and permission changes;

• workspace settings;

• administrative actions;

• security alerts.

An AI workflow may create additional events:

• prompts and model invocations;

• uploaded files and connector access;

• retrieval of documents or records;

• model-provider routing or fallback use;

• tool or agent execution;

• generated outputs and external sharing;

• human review or support access;

• retention and policy-setting changes;

• blocked, failed, overridden, or retried actions.

If those events sit outside the logging scope, the buyer may have an enterprise log without having an AI activity record.

The feature may be real.

The incident-reconstruction capability may still be incomplete.

What it actually proves

A current, product- and plan-specific audit-log statement may establish that:

• an audit-logging mechanism exists;

• certain identity or administrative events are recorded;

• administrators or vendor personnel can review selected activity;

• logs may be exportable through an API or SIEM integration;

• the vendor maintains operational records for security or troubleshooting.

These are meaningful evidence points.

They establish that logging exists.

They do not identify the complete event taxonomy, field coverage, retention period, customer-access model, export capability, or known exclusions.

Availability is not scope.

A public evidence example

The CodeYourCompliance public evidence profile for Decagon records an audit-logging claim among the public enterprise-readiness evidence observed for Decagon’s AI customer-support platform.

The profile is an evidence index.

It is not verification that customers can access the logs or that the logging scope is sufficient for incident reconstruction.

Decagon’s security page states that tamper-protected logs capture key events ranging from logins to unusual activity. The same page says access to those logs is limited to senior engineering leadership.

A separate security and compliance page states that Decagon maintains audit logs of activity, errors, and warnings on production systems.

These statements provide useful evidence that internal logging mechanisms exist.

They do not establish that enterprise customers receive a customer-facing audit trail.

The public materials do not identify:

• the complete event taxonomy;

• which AI-specific actions are recorded;

• whether customer administrators can access or export the records;

• how long the logs are retained;

• whether prompts, connector retrievals, model routes, tool calls, human access, policy changes, exports, and failed actions are included;

• which important events are explicitly excluded.

The evidence therefore supports a narrow conclusion:

The vendor publicly describes internal audit-logging mechanisms.

It does not establish the event-level visibility available to the buyer for the intended workflow.

What it does not prove

An audit-log claim does not automatically establish:

AI event coverage. The log may not record model invocation, prompt submission, retrieved context, file use, generated output, tool execution, or agent action.

Data-access visibility. The buyer may not be able to see which connector, repository, document, database, or customer record was accessed.

Routing visibility. The log may not identify which model provider, region, deployment, fallback route, or integration handled the request.

Human and administrative access. Support access, abuse review, human evaluation, retention changes, permission changes, connector changes, and model-selection changes may not be visible.

Actor attribution. The record may not distinguish users, administrators, service accounts, vendor personnel, contractors, and automated agents.

Outcome visibility. The event may not show whether an action succeeded, failed, was blocked, was retried, or was overridden.

Evidence usability. The buyer may not know whether timestamps are consistent, fields are documented, records are retained long enough, logs can be exported, or administrators can alter or delete them.

A vendor may have strong authentication and infrastructure logs while leaving the customer-relevant AI workflow largely unobservable.

None of these gaps proves that the events are absent.

They show that the available evidence does not establish whether the buyer can reconstruct them.

Weak-answer pattern

This is the feature-without-event-scope pattern.

The vendor confirms that audit logs exist.

The buyer records:

“Audit logs available.”

But the review file does not identify:

• which events are included;

• which AI-specific events are excluded;

• who can access the records;

• how long they are retained;

• whether customers can export them;

• whether the records support incident reconstruction.

The answer describes the feature.

It does not establish the evidence boundary.

Evidence request

Do not ask only:

“Do you provide audit logs?”

Ask for the event taxonomy.

A stronger request is:

Please provide the audit-event taxonomy for the exact product and plan under review, including AI-specific events, administrative events, human-access events, tool execution, data access, routing, outcome status, and known exclusions.

For each event type, identify:

• actor and actor type;

• timestamp;

• action;

• target object;

• success, failure, block, retry, or override result;

• feature, integration, connector, model, or route involved;

• customer-access model;

• log-retention period;

• export method;

• documentation source.

Then ask:

• Are prompts, model invocations, and generated outputs represented as events?

• Are connector access, retrieved sources, file actions, and tool execution visible?

• Are model routing, fallback use, and agent actions recorded?

• Are support access, human review, and administrative changes logged?

• Can customers export the records, and how long are they retained?

• Which relevant events are explicitly excluded?

The vendor does not need to expose model internals.

It does need to show whether the buyer can reconstruct the events that matter for the intended use case.

Review note

A weak review note says:

“Audit logs available.”

A stronger review note says:

The vendor publicly describes audit-logging mechanisms, but the current evidence does not establish customer access, export capability, retention, or coverage of AI-specific events such as model invocation, connector access, routing, tool execution, human review, output sharing, and policy changes. The response is not evidence-complete for incident reconstruction or accountability in the intended workflow.

That note does not say the vendor lacks logging.

It says the buyer does not yet know whether the available logs support the decision being made.

Usage boundary

The required log scope depends on what the AI system is allowed to do.

A drafting assistant used with low-sensitivity internal content may require less event detail than an agent that accesses customer records, executes tools, changes systems, or produces external communications.

Until the relevant event scope is evidenced:

Limit use to workflows where missing audit events would not prevent incident investigation, accountability, or reversal. Do not rely on the vendor’s audit logs for high-impact actions, sensitive data access, autonomous execution, system changes, or external decisions until event coverage, customer access, retention, export, and known exclusions are confirmed.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

Audit-log availability can support that review.

It cannot replace event-level scope.

A log that omits the decisive event cannot support the conclusion the buyer expects from it.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRiskAssessment #AuditLogs #AIGovernance

2026-07-29: PASS
2026-07-30: FAIL

The first question is whether the host changed or the test changed. The record cannot answer it.

The package inventory may have moved, but the baseline, freshness rule, policy or evaluator may also have changed. The same evidence can produce a different result when it is judged under a different evaluation context.

With only PASS and FAIL, there is no way to separate a change in the system from a change in the decision conditions. The conclusions were retained. The basis for them was not.

That is why a stored control result is not replayable evidence. An earlier note asked whether audit evidence can survive replay; the harder implementation question is what must be retained for that replay to mean anything.

What must survive the original decision

Replayable evidence is not the same as retained documentation. Replay does not mean running the same scanner again.

A new scan observes the system as it exists now. Replay reconstructs a previous decision from the artifacts that existed when that decision was made.

For that reconstruction to be credible, the retained package must include more than the observation. It must also preserve the schema that admitted the evidence, the context that supplied the threshold or baseline, the policy artifact, the selected decision engine, the implementation version and an integrity digest binding the assessment to the original evidence bytes.

Each artifact answers a different review question.

The evidence shows what the collector observed. The schema shows what input was considered admissible. The context records the rule parameters used at that time. The policy contains the decision logic. Runtime metadata identifies the execution path. The digest shows whether the retained evidence has changed since evaluation.

If any of these elements is missing, part of the decision must be inferred rather than replayed.

A timestamped result may still be useful operationally, but it is only a historical claim unless the decision path can be reconstructed.

The collector should not decide compliance

Consider a read-only collector operating within an explicit audit boundary. It returns the following package inventory:

{
  "packages": [
    {"name": "openssl", "version": "3.0.13"},
    {"name": "python3", "version": "3.12.3"},
    {"name": "curl", "version": "8.5.0"}
  ]
}

This tells us what the collector observed. It does not tell us whether the host complies with a package baseline.

That decision depends on a separate context:

{
  "required_packages": [
    {"name": "openssl", "version": "3.0.13"},
    {"name": "python3", "version": "3.12.3"},
    {"name": "curl", "version": "8.5.0"}
  ],
  "unexpected_package_mode": "ignore"
}

If the required OpenSSL version changes, the same inventory may fail. If unexpected packages are changed from ignore to fail, an additional package may alter the result even though nothing on the host has changed.

The observation and the assessment therefore belong to different layers.

Package names and versions come from collection. Missing packages, mismatched versions and unexpected packages are derived by comparison. Pass or fail is produced only after policy evaluation.

Writing the control result back into the evidence object hides those transitions. The final document becomes easier to read, but harder to inspect. A reviewer can see the conclusion but cannot tell which facts came from the host, which were derived by the replay engine and which depended on policy choices.

Invalid, stale and failed are different states

A control should fail only after admissible evidence reaches the policy and the required condition is not met.

If the evidence does not satisfy its schema, the policy should not evaluate it. A missing timestamp, malformed package entry or incorrect field type is a problem with the evidence path, not proof that the host failed the control.

The assessment should record that distinction:

evidence_status = invalid_evidence
control_status = unknown
decision_executed = false

Evidence can remain intact and still become too stale to support a current conclusion. In this control, stale evidence creates a different problem from either invalid evidence or policy failure. A package inventory may be structurally valid and still be too old for the decision being attempted.

Suppose the context requires evidence collected within the previous 24 hours. An inventory from three days ago cannot support a current package-baseline decision. It does not prove that the host is non-compliant. It proves that the available observation is no longer fresh enough to answer the question.

evidence_status = stale_evidence
control_status = unknown
decision_executed = false

These distinctions are not cosmetic. They imply different actions.

A failed control may require investigation of the host or baseline. Stale evidence requires recollection. Invalid evidence requires examination of the collector, schema or processing pipeline.

A dashboard that reduces all three conditions to red has removed information that operations and audit both need.

Collection and policy execution are separate boundaries

The earlier implementation note moved from evidence principles to an executable control. The public reference artifacts now apply that structure through a fixed replay sequence:

evidence schema gate

-> context schema gate

-> integrity gate

-> freshness gate

-> derived facts

-> policy evaluation

-> assessment output

For the package-baseline example, the comparison stage derives three groups of facts:

• missing_packages

• version_mismatches

• unexpected_packages

The policy then decides how those facts affect the control result.

A missing required package causes failure. A version mismatch causes failure. An unexpected package may be ignored or treated as a violation depending on the explicit context.

The collector does not need to know any of this. Its job is to report the inventory accurately and preserve its collection metadata. Policy belongs later in the pipeline, where its input and execution can be inspected independently.

The same boundary applies to the decision engine.

A built-in evaluator can implement the same decision semantics as a Rego policy, but it does not execute the Rego artifact. That difference must appear in the assessment:

builtin:
decision_executed = true
policy_artifact.executed = false

opa:
decision_executed = true
policy_artifact.executed = true

Without this metadata, a result may appear to have been produced by a policy artifact that was never run. The conclusion could be correct while its provenance is false.

Replay should stop before remediation

The package-baseline control can identify a missing package, an incorrect version or an unexpected installation. It does not install, remove or upgrade software.

That is a deliberate boundary.

Remediation requires broader permissions and produces a different class of evidence: authorization records, execution logs, change results and post-remediation observations. Combining those actions with collection makes it harder to establish what state existed before the tool intervened.

A read-only collector preserves the condition it was asked to observe. A separate remediation workflow may act on the assessment later, but it should produce its own evidence package.

Collection establishes what was seen. Correction changes what will be seen next.

Public reference artifacts

The replay structure described here is implemented in the public codeyourcompliance/evidence-validation-pipeline repository.

The current examples are:

• examples/replayable-tls-control

• examples/os-package-baseline

The TLS example applies the pipeline to certificate expiry. The OS package example applies it to required packages, version mismatches and unexpected software.

Both packages retain the evidence, validation schemas, evaluation context, integrity information, policy artifact and execution metadata needed to inspect how an assessment result was produced. Their tests also show how the pipeline behaves when evidence is stale, malformed or modified after hashing.

These examples do not prove that the underlying systems are compliant. They show that a control decision can be preserved in a form that is more inspectable than a stored PASS or FAIL.

They also stop before remediation. Installation, removal, upgrade and other corrective actions belong to a separate workflow with separate permissions and evidence.

Replayability does not establish that a decision was correct. It establishes whether the basis of that decision can still be examined.

Origin and scope

CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• From Evidence Principles to an Executable Control

• Can Your Audit Evidence Survive Replay?

• Why Audit Evidence Goes Stale