It is easy to normalize that result immediately as:

configured = false

Sometimes that is correct.

But sometimes the source was never successfully observed in the first place.

API timeouts, permission problems, incomplete pagination, an unstable or unreachable path between the collector and the source, or even a successful request that did not actually cover the full intended scope can all leave the pipeline with “no object returned.”

Those situations are not the same.

If their collection outcomes are collapsed too early, a failed observation can quietly become negative evidence.

Two empty results that mean different things

The latest CodeYourCompliance example models two collection attempts against the same synthetic target.

In the first scenario, we assume the collector successfully reached the intended source and scope. The query completed normally, the result set was complete, and no matching legacy TLS configuration object was found.

The modeled result is:

collection_status = success
result_completeness = complete
observed_state = observed_absent
absence_assertion_supported = true

In the second scenario, the same intended query timed out before the source state was obtained:

collection_status = failed
result_completeness = not_observed
observed_state = unobserved
absence_assertion_supported = false

Neither scenario returns a matching object.

Only the first supports an assertion that the object was absent.

They describe two different observation states, so normalization should not erase that difference.

Absence is still an observation

Negative evidence is easy to describe as “nothing was there.”

That is too loose.

To state that something was absent, the collection path still needs to show what was observed, where it was observed, how it was queried, and whether the observation actually completed.

Not seeing something and confirming its absence are different things.

A bounded absence statement might look like this:

At timestamp T,
through query X,
against synthetic-endpoint-443,
no matching legacy TLS protocol configuration was observed.

That is not the same as:

Legacy TLS is disabled.

The first statement describes an observation.

The second is already a configuration conclusion and may later become part of a control conclusion.

Moving from the first to the second may require more evidence or policy logic.

HTTP 200 is not enough

The first version of the synthetic example exposed an important problem during adversarial review.

It was too easy to read:

HTTP 200
+ empty result

as sufficient proof that the object was absent.

That is too strong.

These responses can look perfectly normal without proving that the intended observation was complete.

The revised synthetic example therefore makes several assumptions explicit. For the observed_absent scenario, we assume by construction that:

• the intended source is authoritative for the synthetic condition

• the intended scope was covered

• the query semantics are defined

• pagination is complete

• the collecting identity has sufficient visibility over the declared scope

These are scenario assumptions.

They are not runtime validations.

The example therefore does not claim that:

HTTP 200 + empty payload

generally means absence.

Collection status belongs in the evidence chain

The new Observed Absence Evidence Contract keeps collection outcome separate from the later control result.

A minimum record can include:

observation_outcome:
  source_ref:
  collection_method:
  collection_attempted:
  collection_status:
  query_semantics_ref:
  scope_ref:
  scope_reached:
  response_status:
  result_completeness:
  observed_state:
  absence_assertion_supported:
  failure_reason:
  observed_at_utc:

The point is not to preserve every transport detail.

The point is to stop materially different collection outcomes from being normalized into the same fact.

At minimum, the pipeline may need to distinguish states such as:

present
observed_absent
unobserved
indeterminate

observed_absent means the observation completed and the query semantics support an absence statement.

unobserved means the intended source state was never successfully obtained.

A timeout belongs in the second state. It should not be converted into false.

A failed observation should stay failed

Suppose an API request times out.

The collector does know one thing: the collection attempt failed.

It does not know whether the configuration object exists.

A more defensible record is therefore:

collection_status = failed
observed_state = unobserved
absence_assertion_supported = false

not:

configured = false

This becomes important once the evidence reaches policy evaluation.

If unobserved is converted into false, downstream logic can no longer distinguish between:

the condition was observed to be absent

and:

we never successfully checked it

And that is how a collection problem quietly turns into a claim about the target state.

This happens before evidence sufficiency

Last week’s control-to-evidence mapping work asked whether all required proof inputs had been assembled before policy evaluation.

This week’s problem happens earlier.

Before the pipeline can ask whether the evidence set is complete, it first has to know what each collection attempt actually produced.

The path is closer to:

source system
-> collection attempt
-> observation outcome
-> raw evidence
-> transformation
-> normalized fact
-> evidence admissibility
-> evidence sufficiency
-> policy evaluation
-> control result

An unobserved outcome may eventually leave an evidence requirement unsatisfied.

An observed_absent outcome may itself become valid evidence for a particular requirement.

But those decisions should happen after the collection outcome has been preserved, not after both states have already been flattened into the same Boolean value.

Observed absence still does not mean PASS

There is another boundary.

Even when observed_absent is recorded correctly, it does not mean the control automatically passes.

Suppose the collector confirms that no legacy TLS configuration object was observed.

Whether that absence supports the control still depends on the control design, source authority, evidence mapping, and policy semantics.

The collection layer should make only the narrower statement it can support.

It should not make the final control decision.

For that reason, the synthetic comparison stops before control evaluation.

At that point it models only the observation outcome.

Public reference artifacts

The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline repository:

• Observed Absence Evidence Contract

• Observed Absence vs Collection Failure Example

• Current Implementation Status

The current artifact models collection-outcome semantics through synthetic examples.

It does not yet validate query semantics, pagination completeness, permission visibility, or source authority automatically.

It also does not determine whether an observed absence is sufficient to satisfy a real control requirement.

Origin and scope

CodeYourCompliance

Website: Compliance Evidence Automation | CodeYourCompliance

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• The Collector Is Part of the Evidence

• Invalid Evidence Is Not a Failed Control

• A Control Result Is Not Replayable Evidence

The first is present. Its schema is valid, integrity is verified, freshness is acceptable, and the required provenance is available.

The second is missing.

A pipeline that checks only the evidence it received may still continue to control evaluation. Every object in front of it looks valid, but the proof set is incomplete.

Nothing is wrong with the first evidence object. There is simply not enough evidence to support the declared control claim.

That is a different problem from invalid evidence.

Admissibility applies to an evidence object

The evidence-admissibility gate answers whether an individual object can be used.

For example:

schema = valid
integrity = verified
freshness = fresh
provenance = present

Those checks say something about the quality of that object.

They do not establish that every evidence requirement for the control has been satisfied.

A control can therefore reach this state:

present evidence = admissible
required evidence set = incomplete
control decision = blocked
control status = unknown

The missing evidence should remain missing. It should not be converted into either a pass or a failure.

This is the boundary introduced in the latest CodeYourCompliance artifact: object-level admissibility and evidence-set sufficiency are separate checks.

A simple TLS example

The public Control-to-Evidence Mapping Example uses a synthetic TLS control.

The declared control requires two evidence types:

tls_certificate_state
tls_runtime_protocol_state

In the complete scenario, both requirements are modeled as satisfied.

The resulting state is:

evidence_set_status = complete
policy_evaluation_allowed = true
control_status = not_evaluated

The example deliberately stops before control evaluation. A complete evidence set means the downstream policy is allowed to run. It does not mean the control has passed.

The incomplete scenario contains certificate-state evidence but no runtime protocol-state evidence:

evidence_set_status = incomplete
policy_evaluation_allowed = false
control_status = unknown

The certificate evidence remains usable. The absent protocol evidence does not become a failed control condition.

The examples are synthetic. They do not execute an admissibility gate, sufficiency gate, downstream control policy, or OPA. The recorded states are modeled scenario assertions used to make the boundary inspectable.

The control needs an explicit evidence map

The Control-to-Evidence Mapping Contract records which evidence requirements belong to a control before policy evaluation starts.

A minimal structure includes:

control_evidence_mapping:
  mapping_id:
  mapping_version:
  control_id:
  control_version:
  control_claim:
  evaluation_rule:
  evidence_requirements:
    - requirement_id:
      requirement_version:
      required:
      evidence_type:
      evidence_layer:
      minimum_objects:
      scope_ref:
      admissibility_required:

The useful part is the relationship between the control claim and its required proof inputs.

For each requirement, the assessment should be able to show what evidence type was expected, which scope it applied to, whether admissibility was required, and whether the requirement was actually satisfied.

Without that mapping, a pipeline can easily confuse:

everything received was valid

with:

everything required was received

Those statements are not equivalent.

Missing evidence should stay unresolved

Missing evidence is easy to mishandle because a binary control model wants a binary answer.

Consider an API timeout while collecting one required evidence type.

The timeout does not establish that the control condition failed. It establishes that the required observation was not obtained.

The state should remain explicit:

required evidence missing
-> evidence set incomplete
-> control evaluation blocked
-> control status unknown

This keeps collection failure separate from control failure.

The same applies when a source is unreachable, an expected object is absent, or the declared scope was only partially observed. Those conditions may require recollection or review, but they are not themselves proof that the target violated the control.

The mapping is part of replay history

The evidence map can change over time.

Suppose mapping version 0.1.0 requires certificate-state evidence only.

A later version adds runtime protocol-state evidence.

If an old assessment is replayed using the new mapping, it is being evaluated against a different proof requirement.

The historical path therefore includes more than evidence and policy:

control claim
-> evidence requirement map
-> admissible evidence set
-> policy input
-> policy evaluation
-> control result

Preserving the historical policy while discarding the historical evidence map still changes the question being replayed.

This is similar to policy versioning, but it occurs earlier in the pipeline. The evidence map determines what must exist before the policy is allowed to run.

Sufficiency still does not prove control correctness

A complete evidence set only proves completeness against the declared mapping.

The mapping itself may still be wrong.

It may omit an important evidence type. The declared scope may be too narrow. A requirement may point to an inappropriate source. A technically valid evidence object may not be the right proof for the claim being assessed.

The mapping therefore answers a limited question:

Did the declared evidence requirements get satisfied?

It does not establish that those were the correct requirements.

That remains a control-design and review problem.

Where this sits in the pipeline

With this addition, the current evidence path becomes:

source system
-> collector
-> raw observation
-> transformation
-> normalized fact
-> evidence admissibility
-> evidence sufficiency
-> policy input
-> policy evaluation
-> control result

Previous artifacts dealt with whether evidence survives collection, transformation and evaluation with enough provenance to be replayed.

This artifact adds a different check before policy execution: whether the declared proof set is actually complete.

A valid evidence object can still be insufficient for the claim being made.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM prescribes this mapping schema, the TLS example, an evidence-sufficiency gate, or OPA.

The engineering interpretation is narrower.

Before producing a control conclusion, the evidence package should preserve which proof inputs were required, which were present, which were admissible, and which remained missing.

Otherwise a valid piece of evidence can be used to support a conclusion that required more proof than the pipeline collected.

Public reference artifacts

The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline repository:

• Control-to-Evidence Mapping Contract

• Control-to-Evidence Mapping Synthetic Example

• Evidence Admissibility Gate

The current artifact models evidence sufficiency through a contract and synthetic examples.

It does not yet implement automatic evidence-set assembly, repo-wide evidence-requirement schema enforcement, control-catalog version governance, or cryptographic binding of a mapping artifact to an assessment result.

Origin and scope

CodeYourCompliance

Website: Compliance Evidence Automation | CodeYourCompliance

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• Invalid Evidence Is Not a Failed Control

• A Control Result Is Not Replayable Evidence

• From Evidence Principles to an Executable Control

The usual explanation is that something changed in the target system. That is possible, but it is not the only explanation. The evidence may be unchanged. The normalized facts may be unchanged. The evaluation context may also be unchanged.

The policy may have changed instead.

If the assessment history preserves only:

previous result: PASS
current result: FAIL

then the difference is visible, but its cause is not.

That becomes an evidence problem before it becomes a control problem.

The result depends on the policy that produced it

Policy-as-code makes evaluation repeatable. It does not make the result independent of the rule being evaluated.

The synthetic example in the public Policy Evaluation Provenance Example keeps one policy input constant:

days_to_expiry = 30
minimum_certificate_days_remaining = 30

Policy version 0.1.0 evaluates:

days_to_expiry >= minimum_certificate_days_remaining

The modeled result is PASS.

Policy version 0.2.0 evaluates:

days_to_expiry > minimum_certificate_days_remaining

The modeled result is FAIL.

The target-state identifier, normalized fact and evaluation context are held constant by construction. Only the policy semantics change.

The example does not prove that a real target remained unchanged between two assessments. It isolates one variable so that the decision difference can be attributed to the policy change inside the synthetic scenario.

The useful boundary is therefore narrower:

same policy input
+ different policy semantics
-> different decision

result difference alone
does not establish target-state change

A changed result needs more context before it can be called drift.

Policy version is only one part of evaluation provenance

Recording a field such as:

policy_version = 0.2.0

is useful, but replay needs more than a version label.

The Policy Evaluation Provenance Contract models the decision path with fields such as:

evaluation:
  engine:
  engine_version:
  policy_id:
  policy_version:
  rule_id:
  rule_version:
  policy_artifact_ref:
  policy_artifact_sha256:
  policy_input_ref:
  policy_input_sha256:
  evaluation_context_ref:
  evaluation_context_sha256:
  implementation_version:
  evaluated_at_utc:
  decision_executed:
  result:

The point is not to collect metadata for its own sake.

A reviewer needs to know which policy artifact was evaluated, which input it consumed, which context was applied, which decision entry point ran, and which engine and surrounding implementation produced the result.

Without that information, a historical assessment can easily be replayed using today’s policy instead of the policy that produced the original decision.

The evidence may be the same. The replay is not.

A policy hash does not establish policy authority

The policy artifact can also be hashed.

That improves identity. A SHA256 digest can show which policy bytes were referenced by an evaluation record and whether the preserved artifact later changed.

It does not show whether those bytes should have been used.

This distinction matters:

policy byte identity
!=
policy release authority

A digest does not prove that the policy was approved, current at the time of evaluation, applicable to the assessed scope, or released by an authorised control owner.

A replay package can therefore reproduce a decision against the wrong policy artifact with perfect integrity.

The replay is technically faithful. The policy selection may still be wrong.

Signed policy bundles, approval records and trusted policy-release provenance are separate concerns.

Deterministic evaluation does not validate the policy

OPA can evaluate the same input against the same Rego artifact and return the same result repeatedly.

That proves something useful: the identified evaluation path is deterministic.

It does not prove that the policy represents the correct control interpretation.

A threshold can be wrong. An exception may be missing. A rule may be valid for one system scope and unsuitable for another.

Evaluation provenance answers:

What policy path produced this result?

It does not answer:

Was this the correct policy for this control?

That judgment remains outside the policy engine.

Replay has to preserve the historical decision path

This repository already contains a bounded Replayable TLS Control.

Its pipeline is:

evidence schema
-> context schema
-> integrity
-> freshness
-> derived facts
-> policy evaluation
-> assessment result

That reference implementation records hashes for evidence, schemas, evaluation context and policy, together with implementation and runtime provenance.

The new policy-evaluation provenance artifact does not replace that implementation. It extracts one specific rule from it:

A historical control result should remain tied to the policy and evaluation path that produced it.

That makes this problem slightly different from the earlier question of whether a control result is replayable at all.

The first question is:

Can I reproduce this result?

The next question is:

Am I reproducing the historical decision,
or re-evaluating historical evidence
under a different policy?

Both operations may be useful.

They should not be confused.

The decision path has more than one moving part

Once collection, normalization and policy evaluation are separated, a changed assessment result can have several explanations:

target changed
collector changed
transformation changed
policy input changed
evaluation context changed
policy changed
engine or implementation changed

A PASS -> FAIL transition does not identify which one occurred.

The control result is the end of the pipeline. It is not a complete explanation of the pipeline.

Public reference artifacts

The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline:

• Policy Evaluation Provenance Contract

• Policy Evaluation Provenance Synthetic Example

• Replayable TLS Control

The synthetic comparison does not claim to be an OPA runtime capture. The repository also does not yet implement signed policy bundles, trusted policy-release attestation or generic cross-policy regression testing.

Those remain separate implementation steps.

A control result is downstream of evidence, context and policy.

If the policy path disappears, part of the decision history disappears with it.

Origin and scope

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• A Control Result Is Not Replayable Evidence

• Invalid Evidence Is Not a Failed Control

• From Evidence Principles to an Executable Control

The first ten notes focused mainly on vendor claims and the evidence behind them.

There is another evidence problem once the documentation looks complete.

A vendor may have a policy, workflow diagram, product documentation, and configuration showing how an AI agent is expected to behave.

That does not show whether the control is enforced when the agent actually runs.

Claim

The documentation says:

“High-risk actions require human approval.”

The policy defines the requirement.

The workflow contains an approval step.

The product may provide a setting for it.

These are useful evidence of the intended control.

They do not show that every qualifying action is actually stopped and sent for approval.

Why it sounds sufficient

Policies and configuration are familiar review artifacts.

If the approval rule is documented and the feature is enabled, it is easy to record human oversight as an implemented control.

The problem is the execution path.

An agent may select tools, call APIs, retry failed actions, or reach the same result through a different sequence.

A reviewer therefore still needs to know:

• what triggers approval;

• whether execution stops before the action;

• what happens after rejection;

• how retries are handled;

• whether another tool path can avoid the trigger;

• what happens if the approval mechanism fails;

• whether the approval can be linked to the final action.

Those questions cannot be answered by the policy alone.

What it actually proves

A policy can show that the control was defined.

Product documentation can show that an approval mechanism exists.

Configuration evidence can show that it was enabled.

These support control design and configuration.

Evidence of enforcement comes from somewhere else.

For example, an execution record could show:

Trigger: The agent attempted an action requiring review.

Pause: The action stopped before execution.

Reviewer: The request was sent to an attributable reviewer.

Decision: Approval or rejection was recorded.

Execution: The action occurred only after approval.

Trace: The approval event and final action can be linked.

That is evidence about what happened, not only what was supposed to happen.

A public evidence example

The CodeYourCompliance Trust Signal profile for Decagon shows the distinction at the public-evidence layer.

The profile identifies security and AI data-usage surfaces, along with enterprise evidence such as audit logs and AI guardrails. It also keeps an explicit boundary: those public security materials are review surfaces, not implementation verification.

That boundary becomes important for agent controls.

Trust Signal can help locate the documented evidence surface.

It cannot show, by itself, whether a guardrail or approval control was enforced in a particular execution.

What it does not prove

A documented approval control does not automatically establish:

Trigger coverage. The rule may exist without showing that every relevant action is detected.

Bypass resistance. Another tool or workflow may reach the same outcome without triggering approval.

Failure behavior. Documentation may not explain what happens when the approval service fails or times out.

Retry behavior. A rejected action may be attempted again through a different path.

Sequence. An approval record is weak evidence if the action could occur before approval was captured.

Version applicability. Evidence collected against one model, prompt, tool configuration, or agent version may not remain valid after those components change.

Weak-answer pattern

A common answer is:

“Human approval is required for high-risk actions. See our policy and product documentation.”

That may be enough to support the existence and design of the control.

The gap appears when the review stops there.

The buyer has evidence of the rule, but not yet evidence that the rule intercepted the relevant action during execution.

Evidence request

For controls that limit what an agent can do, ask for evidence from the execution path:

Provide evidence showing how actions requiring human approval are identified, paused, presented for review, approved or rejected, and linked to the final execution record. Include rejected actions, retries, alternate paths, and approval-service failure where available.

Useful evidence may include execution traces, approval logs, reviewer identity or role, timestamps, tool-call records, rejected-action records, failure tests, and bypass tests.

The purpose is not to collect more logs.

It is to connect the documented control to the behavior it is meant to constrain.

Review note

Human approval is documented as a control for high-risk actions. The available evidence supports control design, but enforcement across triggering conditions, retries, alternate execution paths, and failure conditions has not yet been demonstrated.

Usage boundary

Until enforcement evidence is available, a documented approval requirement should not be treated as proof that high-impact autonomous actions are consistently intercepted.

For higher-impact use cases, keep the agent within a boundary where failure or bypass of the approval step cannot independently create a consequential action.

The evidence question is no longer only:

What does the policy say?

It is also:

What happened when the control was tested?

Boundary

This material is for evidence structuring and review preparation.

The goal is to make the gap between documented controls and demonstrated behavior visible before those controls are relied upon.

#AIAssurance #AIGovernance #AIAgentTesting

status = not_configured

A normalization layer turns that into:

enabled = false

The second value is easier for a policy engine to consume.

It is also not what the system originally returned.

Someone decided that not_configured should mean false.

That decision happened before policy evaluation.

If the evidence package keeps only the normalized value, the interpretation disappears while its consequence survives.

That is the problem.

Raw observation and normalized fact are different evidence layers

Most systems do not produce evidence in the format a control policy wants.

Commands return text. APIs use vendor-specific fields. Configuration files use their own syntax. One source may say enabled, another 1, another active.

Normalization is necessary.

A typical path looks like this:

raw evidence
→ normalized fact
→ derived fact
→ policy input
→ policy result

The public CodeYourCompliance Evidence Contract already separates these layers.

The distinction is not about data formatting.

It is about where interpretation enters the evidence chain.

The raw observation records what the source returned.

The normalized fact records how the pipeline chose to represent it.

Those are related objects.

They are not the same object.

The same observation can produce different facts

The new public Transformation Provenance Example isolates this problem.

The synthetic raw evidence is held constant:

status = not_configured

Normalization version 0.1.0 applies this rule:

MAP_NOT_CONFIGURED_TO_FALSE

and produces:

enabled = false
state = resolved

Version 0.2.0 uses a different rule:

PRESERVE_NOT_CONFIGURED_AS_UNKNOWN

and produces:

enabled = null
state = unknown

Same raw observation.

Different transformation logic.

Different normalized fact.

The synthetic example deliberately holds the raw observation and target-state identifier constant so that only the transformation changes.

It does not establish which normalization rule is correct.

It establishes a narrower point:

same raw observation
+ transformation difference
→ normalized fact changed
→ target-state change is not established
  by this difference alone

That boundary matters whenever normalized facts are later compared for drift or fed into policy evaluation.

Transformation provenance should travel with the fact

The public Transformation Provenance Contract records the minimum context needed to inspect that path.

The current pattern includes fields such as:

transformation:
  name:
  version:
  rule_id:
  rule_version:
  input_ref:
  input_field:
  input_value:
  output_field:
  output_value:
  output_state:
  executed_at_utc:

This means a reviewer does not only see:

enabled = false

They can also determine which raw observation was consumed, which rule performed the mapping, which implementation version ran, and what output it produced.

That is provenance.

It is not attestation.

Transformation provenance is not transformation correctness

A versioned rule can still be wrong.

A documented mapping can still be unsuitable for a particular control.

A normalizer can contain a bug.

Recording:

rule_version = 0.2.0

does not prove that version 0.2.0 interpreted the source correctly.

It only makes that interpretation identifiable.

This is the same boundary discussed in The Collector Is Part of the Evidence: preserving how something entered the evidence chain is different from proving that the mechanism was correct.

The collector can be wrong.

The transformation can be wrong.

Provenance gives the reviewer something concrete to challenge.

Integrity does not solve the semantic problem

Hashing solves another problem.

Suppose the pipeline records:

source value:
not_configured

normalization:
not_configured → false

integrity:
verified

The integrity check can show that the normalized evidence object has not changed after sealing.

It cannot tell us whether false was the right interpretation in the first place.

A perfectly intact evidence package can preserve a bad transformation.

So these two claims need to remain separate:

integrity verification
!=
semantic correctness

Hashing protects the preserved object.

Transformation provenance explains how that object acquired its meaning.

Policy evaluation starts later than it looks

OPA may receive a clean object such as:

{
  "enabled": false
}

From OPA’s perspective, that input is simple.

From an evidence perspective, it already contains history.

A useful trace is closer to:

control result
→ policy input
→ normalized fact
→ transformation rule
→ raw observation
→ collector
→ source system

This also connects to the admissibility boundary described in Invalid Evidence Is Not a Failed Control.

The policy engine is not where the evidence story begins.

By the time evaluation starts, collection and interpretation have already happened.

If those steps disappear, the final result may still be deterministic while the path that produced its input is not inspectable.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM requires transformation metadata, rule versions, normalization schemas, OPA, or this implementation pattern.

The engineering question is narrower:

Can a reviewer determine how a raw system observation became the fact that was actually evaluated?

If not, the pipeline preserves the control result more carefully than the interpretation that created its input.

Public reference artifacts

The transformation provenance boundary is implemented as a public engineering artifact in the codeyourcompliance/evidence-validation-pipeline repository.

Main references:

• Transformation Provenance Contract

• Transformation Provenance Synthetic Example

• Evidence Contract

The repository currently models transformation provenance through structured metadata and synthetic examples.

It does not yet cryptographically bind transformation or rule digests to normalized facts. It also does not implement automated transformation-version equivalence or regression testing, or repo-wide transformation provenance schema enforcement.

Those are separate implementation steps.

Origin and scope

CodeYourCompliance

Website:

Compliance Evidence Automation | CodeYourCompliance

CodeYourCompliance explores compliance automation, replayable audit evidence, read-only evidence collection, policy-as-code, and evidence packages.

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• The Collector Is Part of the Evidence

• Invalid Evidence Is Not a Failed Control

• Compliance Automation Starts at Evidence

The first nine notes looked at different AI vendor claims: training use, Trust Centers, SOC 2 scope, human review, subprocessors, retention, audit logs, data residency, and questionnaire answers.

The subjects were different.

The review problem was usually the same.

A vendor statement was available, but the supporting evidence was narrower than the statement suggested.

A claim-evidence matrix keeps that difference visible.

Claim

The review file says:

“Vendor assessment completed.”

The questionnaire has been returned.

The Trust Center, DPA, SOC 2 material, subprocessor information, and product documentation have been reviewed.

That shows that a review took place.

It does not show how individual claims were supported.

The file may still contain statements such as:

Customer data is not used for training.

Human review is limited.

Data residency is supported.

Audit logs are available.

Each statement may have a supporting source.

The question is what that source actually covers.

Why it sounds sufficient

Procurement reviews already collect a lot of material.

Adding another matrix can look like duplicate work.

The problem appears when answers and evidence are stored separately.

The questionnaire contains the claim.

The Trust Center contains one source.

The DPA may contain a stronger commitment.

Product documentation may narrow the feature scope.

A SOC 2 report may cover only part of the relevant environment.

If those relationships are not recorded, another reviewer has to reconstruct the reasoning later.

What it actually proves

A claim-evidence matrix records that relationship.

For each material claim, capture:

Claim: What did the vendor state?

Evidence source: What document or page supports it?

Scope: Which product, plan, feature, region, and data type are covered?

Exceptions: What sits outside the statement?

Evidence date: When was the source checked?

Evidence gap: What remains unresolved?

Buyer question: What needs to be clarified next?

Usage boundary: What should remain limited while the gap is open?

The useful sequence is:

vendor claim → evidence source → scope → evidence gap → buyer question → usage boundary

A simple example

Take:

“We do not train on customer data.”

The review should not stop at “No training.”

The matrix may show:

Source: Enterprise data-use documentation.

Scope: The statement applies to a defined commercial product.

Exceptions: Support, abuse monitoring, human review, or opt-in workflows require separate checking.

Evidence gap: Retention, logging, subprocessor handling, and deletion are not established by the training statement.

Buyer question: Which sources establish those remaining data-handling boundaries?

The vendor answer has not changed.

The assumptions around it are now visible.

A public evidence example

The CodeYourCompliance public evidence profile for Cohere separates public evidence surfaces rather than treating the vendor name as one source.

That is useful for locating the source side of the review.

But the profile does not decide whether a particular claim is sufficient for a buyer’s product, plan, use case, data type, region, or contract.

The claim-evidence matrix performs that next step:

evidence surface → claim → scope → remaining gap

Trust Signal helps locate and structure public evidence.

The matrix records what that evidence can actually support in the review.

What it does not prove

The matrix does not prove that a vendor is low risk.

It does not make a source stronger than its scope.

A SOC 2 report can still leave the AI data path unclear.

A subprocessor list can identify third parties without showing actual routing.

A Trust Center can provide useful evidence without resolving the buyer’s use case.

The matrix records those limits.

It does not remove them.

Weak-answer pattern

A common review pattern is:

Claim identified. Supporting document attached. Item closed.

The document may be relevant, but the review file does not record:

• which part of the claim it supports;

• which product or plan is covered;

• whether exceptions exist;

• whether another source is required;

• when the evidence was checked.

The result is a documentation package without a clear evidence chain.

Evidence request

For material claims, ask for the source and its boundary.

For each material AI data-handling, security, privacy, logging, retention, subprocessor, residency, and human-access claim, identify the supporting source, applicable product and plan, covered data types, known exceptions, evidence date, and contractual basis where applicable.

The goal is not more documents.

It is to identify which claims still lack usable evidence.

Review note

Material vendor claims have been mapped to available supporting sources. Some remain partially evidenced because product scope, data-type coverage, exceptions, evidence freshness, or contractual applicability are incomplete.

That is more useful than a spreadsheet containing only Yes, No, and See Trust Center.

Usage boundary

The matrix should always be read in context.

The same claim may support different use depending on the product, plan, data type, region, contract terms, and evidence date.

If the evidence supports only low-sensitivity internal use, record that.

If customer data requires stronger retention or subprocessor evidence, keep that limitation open.

A claim-evidence matrix is not a vendor score.

It is a dated record of what the available evidence supports and what remains unresolved.

The first nine notes focused on vendor statements and supporting sources.

The same question continues after documentation:

What evidence shows that a documented control actually works as described?

Boundary

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample claim-evidence matrix.

#AIVendorRiskAssessment #ThirdPartyRisk #AIGovernance

Six months later, someone reviews the assessment and asks how that expiry date was collected.

Was it read directly from the certificate file? Parsed from command output? Returned by an application API?

If the evidence package does not say, the value may still be intact, but part of its history is missing.

The collector is not just transport between a system and an evidence file. It is part of the chain that produced the observation.

The same field can have different origins

Take a field such as:

certificate_not_after

One collector might read a certificate file directly. Another might call openssl and parse stdout. Another might query an application endpoint.

The final value could be identical in all three cases.

That does not make the collection paths equivalent.

Each method depends on a different source and different interpretation logic. If only the normalized field is retained, those differences disappear.

This becomes more important when results change.

If yesterday’s collector reported one value and today’s collector reports another, the first question is not necessarily whether the target changed.

The collection path may have changed instead.

Record the collector with the observation

The current CodeYourCompliance evidence pattern records collector metadata with the observation:

{
  "collector": {
    "name": "ansible-readonly-tls-collector",
    "version": "0.1.0",
    "mode": "read_only",
    "method": "remote_command_and_file_read",
    "changed_target_system": false
  }
}

The collector name identifies what produced the observation.

The version identifies which implementation was running.

The method records how the state was obtained.

The read-only fields record the intended collection boundary and whether collection changed the target.

The public Collector Provenance Contract extends this with fields such as parser version and source reference.

None of these fields proves that the collector was correct.

They make the production path inspectable.

Read-only does not solve provenance

I have previously treated read-only collection as an audit boundary.

That boundary still matters.

A collector that repairs a system while collecting evidence mixes observation with remediation. Once that happens, it becomes harder to tell whether the recorded state existed before the collector touched the target.

But a collector can be read-only and still produce misleading evidence.

A parser can be wrong.

A configuration path can change.

A missing value can become false when it should remain unknown.

None of those problems requires a write operation.

Two questions therefore remain separate:

The first is about collection restraint.

The second is about provenance.

Collector changes can look like system changes

The public collector provenance example makes this problem explicit.

It contains two synthetic evidence objects for the same service and the same synthetic target-state identifier.

The first uses collector version 0.1.0, reads the configured certificate file and records:

not_after = 2026-09-30T00:00:00Z

The second uses collector version 0.2.0, performs a runtime TLS handshake and records:

not_after = 2026-10-15T00:00:00Z

The target-state identifier is deliberately held constant by the scenario.

The collector version, collection method, parser version and source reference change.

The example does not establish which observation is correct.

It establishes a narrower boundary:

Before a difference is classified as system drift, the collection path needs to remain visible.

This is also why retaining only the final PASS or FAIL is weak. As discussed in Invalid Evidence Is Not a Failed Control, a result only has meaning if the pipeline preserves how that result was reached.

Normalization has the same problem

Raw collection is often only the first step.

A command returns text. A parser extracts values. A normalization layer converts them into stable fields. A policy engine evaluates those fields.

The useful trace is closer to:

Not every implementation detail belongs in the final report.

But the evidence package should retain enough metadata to identify which collection and transformation path produced the evaluated fact.

Otherwise the policy side is visible while the collection side remains opaque.

Provenance is not proof of correctness

There is another boundary worth keeping clear.

Recording collector identity, version and method does not make the collector trustworthy.

A known collector can contain a bug.

A read_only attribute can be wrong.

A documented source can still be unsuitable for the control being evaluated.

Collector metadata provides a trail to inspect.

It does not validate that trail.

Structured provenance improves traceability. It does not turn implementation claims into facts.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM requires Ansible, collector version fields, parser metadata or this evidence model.

The engineering interpretation is narrower.

If system state is being turned into evidence for later control assessment, the collection path should not disappear once the normalized value has been produced.

A reviewer should be able to determine what collected the observation, how it was collected, what source was used and which implementation version was involved.

If that information is missing, part of the evidence history is missing too.

Public reference artifacts

The collector provenance boundary described here is implemented as a public engineering artifact in the codeyourcompliance/evidence-validation-pipeline repository.

Main references:

• Collector Provenance Contract

• Collector Provenance Synthetic Example

• Minimal Evidence Object

The contract defines the provenance fields needed to keep collector identity, version, method, parser and source attached to an observation.

The synthetic example shows why an observation difference should not automatically be classified as target-state drift when the collector also changed.

The repository does not yet cryptographically bind collector implementation digests to evidence objects. It also does not implement automated collector-version equivalence testing.

Those remain separate implementation steps.

Origin and scope

CodeYourCompliance

Website:

Compliance Evidence Automation | CodeYourCompliance

CodeYourCompliance explores compliance automation, replayable audit evidence, read-only evidence collection, policy-as-code, and evidence packages.

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• Read-Only Collection as an Audit Boundary

• Compliance Automation Starts at Evidence

• Invalid Evidence Is Not a Failed Control

A completed questionnaire can still leave the buyer without usable evidence.

AI vendor reviews often ask:

“Do you use customer data for model training?”

“Do you support data residency?”

“Are audit logs available?”

“Do you maintain a subprocessor list?”

The vendor answers.

The questionnaire is marked complete.

The review may still be incomplete.

The question is not only:

Did the vendor answer?

The better question is:

What source supports the answer, what does that source cover, and what remains unresolved?

A questionnaire collects statements.

An evidence review tests whether those statements support the buyer’s intended use.

Claim

The review file says:

“Vendor questionnaire completed.”

Or:

“Vendor confirmed required controls are in place.”

That establishes that the vendor responded.

It does not establish that the answers are evidence-complete.

Why it sounds sufficient

Questionnaires create structure.

They force buyers to ask about security, privacy, retention, subprocessors, model training, audit logs, data residency, and access controls.

That is useful.

The problem starts when the answer becomes the evidence.

A questionnaire may contain:

No customer data is used for training.

SOC 2 Type II available.

Audit logs supported.

Data stored in the United States.

See Trust Center.

Each answer may be accurate.

But none of them tells the buyer, by itself:

• which source supports it;

• which product and plan it covers;

• which data types are included;

• which exceptions apply;

• whether the evidence is current;

• whether the commitment is contractual.

The questionnaire records what the vendor said.

It does not automatically record what the buyer can rely on.

What it actually proves

A completed questionnaire may establish that the vendor responded to a defined set of questions and that relevant claims have been collected.

It can also identify areas that need follow-up.

It does not establish the strength or scope of the evidence behind each answer.

Consider:

Question: Do you use customer data for model training?

Vendor answer: No.

The review file still needs:

Source: Where is that commitment written?

Scope: Which product, plan, endpoint, or feature does it apply to?

Data covered: Prompts, outputs, files, metadata, logs, or all customer content?

Exceptions: Support, abuse monitoring, opt-in evaluation, or other workflows?

Evidence date: When was the source checked?

Contract boundary: Is this public documentation or part of the customer agreement?

Gap: What remains unresolved?

That is the difference between an answer and an evidence record.

A public evidence example

The CodeYourCompliance public evidence profile for Cohere shows how a vendor review can move beyond a questionnaire answer.

The profile separates public evidence surfaces rather than treating the company name as one source.

A response such as:

“SOC 2 available.”

or:

“Customer data is not used for training.”

still needs to be mapped to the specific source that supports it.

The reviewer then needs to check the product and plan scope, evidence date, data types covered, known exceptions, and whether a stronger contractual or customer-specific source is required.

The profile helps locate evidence surfaces.

It does not make the review conclusion.

That distinction matters.

Questionnaire answer → evidence source → scope → evidence gap

That is evidence conversion.

What it does not prove

A completed questionnaire does not automatically establish:

Source strength. “Yes,” “No,” or “See Trust Center” does not show whether the source is contractual, audited, product documentation, marketing material, or unsupported.

Product scope. A company-level answer may not apply to the exact product, plan, feature, endpoint, or integration.

Data-path scope. A training answer does not establish retention. A residency answer does not establish processing location. A subprocessor answer does not establish routing.

Freshness and exceptions. Supporting documents may change, and binary answers may hide support access, safety review, fallback providers, beta features, or configurable behaviour.

Contract applicability. Public documentation may not create the same commitment as a DPA, order form, or product addendum.

None of these gaps means the vendor gave a false answer.

They mean the answer still needs evidence mapping.

Weak-answer pattern

This is the answer-without-source pattern.

The buyer asks:

“Are audit logs available?”

The vendor answers:

“Yes.”

The spreadsheet becomes green.

But the review file does not identify which events are logged, which plan is covered, how long logs are retained, whether customers can export them, or whether AI-specific activity is visible.

The questionnaire looks complete.

The evidence record is not.

Evidence request

Do not ask the vendor to complete another questionnaire.

Ask the vendor to support the existing answers.

For each material AI, security, privacy, retention, logging, subprocessor, data-residency, and human-access claim, identify the supporting source, applicable product and plan, relevant data types, known exceptions, evidence date, and contractual basis where applicable.

For higher-impact claims, ask:

• Does the source apply to the exact product and feature?

• Which data types are explicitly covered?

• Which exceptions exist?

• Has the supporting evidence changed?

• Is stronger private or customer-specific evidence required?

The goal is not more questions.

The goal is claim-to-evidence mapping.

Review note

The vendor completed the requested questionnaire, but several material responses have not yet been mapped to product-specific supporting evidence. Source strength, product and plan scope, data-type coverage, exceptions, evidence freshness, and contractual applicability remain incomplete for key AI data-handling claims.

The questionnaire is complete.

The evidence mapping is not.

Usage boundary

Until material answers are mapped to evidence:

Limit use to low-sensitivity internal or test data. Do not expand to customer-confidential data, regulated data, source code, employee records, autonomous actions, or externally relied-upon outputs where unsupported questionnaire answers would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit remains:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A questionnaire can collect claims.

It cannot determine whether those claims are supported.

The useful sequence is:

vendor answer → evidence source → scope → evidence gap → buyer question → usage boundary

A completed questionnaire is an input.

It is not an evidence-complete review.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRiskAssessment #ThirdPartyRisk #AIGovernance

The collection partially fails.

The output file still exists.

A downstream policy engine reads the file, cannot find the expected field, and returns fail.

The report now says the control failed.

But the evidence may have failed instead.

That distinction has to survive the pipeline.

If it does not, compliance automation can manufacture control failures from collection failures, integrity failures, or invalid evaluation context.

The audit problem starts before the policy engine.

A file is not automatically admissible evidence

A common pipeline looks like this:

collect
  ↓
evaluate
  ↓
pass / fail
  ↓
report

The hidden assumption is that collection produced usable evidence.

That assumption is unsafe.

An artifact can exist while still being unusable. Its schema may be invalid, its integrity check may fail, its freshness window may have expired, or the evaluation context may be malformed.

None of those conditions proves that the control failed.

They mean the system lacks a defensible basis for a control decision.

That requires an explicit gate.

Evidence admissibility is not control evaluation

The pipeline should answer two different questions.

First:

Is this evidence admissible for evaluation?

Then:

Does the observed condition satisfy the control policy?

Those questions should not share the same result field.

The CodeYourCompliance reference implementation uses this sequence:

evidence schema gate
→ context schema gate
→ integrity gate
→ freshness gate
→ derived facts
→ decision evaluation
→ assessment result

If an earlier gate fails, the control decision does not execute.

The state remains:

decision_executed = false
control_status = unknown

That is not a failed control.

It is a blocked decision.

Invalid evidence should stop the pipeline

Take an integrity mismatch.

The evidence package no longer matches the sealed object the pipeline expected.

The wrong transition is:

invalid_evidence
→ control_status = fail

The correct transition is:

invalid_evidence
→ decision_executed = false
→ control_status = unknown

The policy engine has nothing admissible to evaluate.

Returning fail would create a control conclusion that the evidence does not support.

The implementation rule is simple:

No admissible evidence.
No control decision.

Stale evidence uses the same boundary

Freshness is a different failure mode, but the same gate applies.

Evidence may still have a valid integrity hash and known provenance while being too old for the current evaluation window.

That evidence is not corrupt.

It is stale.

A freshness policy may classify it as stale_evidence, but that classification belongs to evidence admissibility.

It is not the downstream control decision.

This is why the public example separates:

admissibility_evaluation

from:

control_decision

The control decision remains unexecuted and unknown.

Evidence evaluation is not control evaluation.

OPA is not the boundary

OPA can evaluate either kind of policy.

A Rego freshness policy can decide whether evidence may proceed. A control policy decides whether admissible facts satisfy a declared condition.

The tool is not the point.

The stage is.

Those decisions should remain separate even if both are implemented in OPA.

The state transition should be explicit

The useful rule is not a longer list of statuses.

It is a prohibited transition.

This should never happen:

invalid_evidence
→ control_status = fail

Neither should this:

stale_evidence
→ control_status = fail

The allowed path is narrower:

admissible evidence
→ decision executed
→ pass / fail

Everything before that belongs to evidence qualification.

fail means the control policy evaluated admissible evidence and found the condition unsatisfied.

unknown means the pipeline could not reach that judgment.

Reports should preserve the blocked decision

A reporting layer should not translate every non-pass state into non-compliant.

It should preserve the failed gate, evidence status, whether the decision executed, and the control status.

Reports summarize a decision path.

They should not invent one.

MAS TRM-inspired means engineering interpretation

This is not a claim that MAS TRM prescribes this state machine, OPA, Rego, SHA256, or this implementation.

The engineering question is narrower:

Can the evidence pipeline distinguish failure of the control from failure of the material used to assess it?

If not, the final report may look deterministic while its underlying decision path is not.

A failed control is a control conclusion.

Invalid evidence is the absence of a defensible basis for that conclusion.

That is the boundary.

Public reference artifacts

The admissibility boundary described here is implemented in the public codeyourcompliance/evidence-validation-pipeline repository.

The main reference package is:

• examples/replayable-tls-control

• docs/evidence-admissibility-gate.md

The TLS example separates evidence validation from control decision evaluation. Schema validation, integrity verification, freshness checks and evaluation-context validation occur before the control decision.

If a gate fails, the decision does not execute. The result remains unknown, not fail.

These examples do not prove that the underlying system is compliant. They show whether the pipeline had a defensible basis to make a control decision at all.

They also stop before remediation. Certificate renewal, configuration changes and other corrective actions belong to a separate workflow with separate permissions and evidence.

Origin and scope

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• From Evidence Principles to an Executable Control

• Can Your Audit Evidence Survive Replay?

• Why Audit Evidence Goes Stale

The remediation owner says the work is complete.

There is evidence.

But does that evidence actually support the closure requirements?

That is the problem in the first Awesome GRC Workflows case.

Evidence attached ≠ requirement satisfied

A document existing in the package tells you very little by itself.

Take a test result.

The useful questions are not:

Is there a test document?

They are:

What requirement is this supposed to support?

What was actually tested?

When was it tested?

What does the result really establish?

The same applies to other artifacts.

An evidence inventory may say a procedure exists.

That does not mean you have reviewed the procedure.

A management statement may provide context.

That does not turn it into independent verification.

A deployment record may show that a change happened.

That does not automatically prove sustained effectiveness.

The reviewer has to connect:

requirement
→ evidence
→ limitation
→ conclusion

That is the work.

The case

In Remediation Evidence Closure Review, you act as an Evidence Review Analyst.

You receive a fictional remediation closure package containing documented requirements and multiple forms of synthetic evidence.

Your task is to review the submitted package and determine what the evidence can actually support.

You produce three deliverables:

1. Evidence-to-requirement review workpaper

2. Evidence deficiency register

3. Closure-readiness memo

This is not a quiz.

You have to inspect the records, identify gaps and contradictions, document what remains unresolved, and leave behind work another reviewer can follow.

Recommendation ≠ approval

There is another boundary in the exercise.

You may analyze the evidence.

You may identify deficiencies.

You may request additional evidence.

You may prepare a closure-readiness recommendation.

You may not close the issue.

You may not accept residual risk.

You may not make legal determinations or execute remediation.

Correct analysis does not create approval authority.

That distinction is part of the work sample.

Run the case

The case is synthetic. The organization, records, systems, people, and evidence are fictional or safely constructed.

Start with the README, scenario, task brief, inputs, and templates.

Complete the three deliverables before opening the reviewer guide or reference answer.

Your job is not to guess what happened.

Your job is to determine what the submitted evidence can actually support.

Run the case.

https://github.com/codeyourcompliance/awesome-grc-workflows/tree/main/cases/remediation-evidence-closure-review

A country name sounds like a boundary.

“Customer data is stored in the United States.”

“EU data residency is available.”

“Enterprise customers can select a region.”

These statements feel specific.

They identify a place, and a place can make the data path appear settled.

But in an AI vendor review, location is only the beginning of the question.

A residency statement may describe where one category of data is stored while saying little about where the rest of the workflow is processed, cached, backed up, reviewed, or routed.

The review question is not only:

Where is customer data stored?

The better question is:

Which data is covered, what happens to it, which systems and providers receive it, and which parts of that path are actually restricted to the stated region?

A hosting location can be accurate while the geographic boundary remains incomplete.

That is the evidence gap.

Claim

The vendor says:

“Customer data is stored in the United States.”

Or:

“EU data residency is available.”

Or:

“Enterprise customers can select a hosting region.”

These statements may establish that defined customer data is stored within a stated location.

They do not establish the complete geographic path of the buyer’s data.

Why it sounds sufficient

Data residency is often reduced to a country, region, or dropdown selection.

That works well in a questionnaire.

The buyer asks where customer data is hosted. The vendor names a location. The answer appears complete.

But an AI workflow rarely involves one database and one data object.

A single interaction may create:

• prompts and outputs;

• uploaded files;

• audio and transcripts;

• embeddings and indexes;

• metadata and usage records;

• logs and moderation events;

• support artifacts;

• cached copies;

• backups and recovery copies.

The application may store customer records in one region while a model provider, transcription service, analytics platform, support tool, or security service operates through a different path.

The storage location may be clear.

The processing location may not be.

The primary database may sit in one country.

Backups, logs, support systems, local caches, or downstream providers may follow different rules.

This is how a true statement can create the wrong impression.

“Stored in the United States” may accurately describe the primary application environment.

It may not describe the entire service.

What it actually proves

A current, product-specific residency statement may establish that:

• a primary hosting region has been identified;

• defined customer content is stored in a stated location;

• a regional storage option exists;

• the vendor has documented a geographic control;

• a particular plan or deployment supports regional selection.

These are meaningful evidence points.

They establish something about location.

They do not automatically establish:

• which data types are covered;

• whether storage and processing occur in the same region;

• where external providers operate;

• where backups and logs remain;

• where support personnel may access data;

• whether exceptions alter the path;

• whether the commitment is contractual;

• whether the same boundary applies to every product, plan, and feature.

Storage is part of the data path.

It is not the whole data path.

A public evidence example

The CodeYourCompliance public evidence profile for Granola records a US data-residency reference among the public evidence surfaces observed for Granola’s AI meeting-notes product.

The profile was last checked on 27 June 2026. It also records that regional residency outside the United States was not observed during that review.

The profile is an evidence index.

It is not implementation verification, a security assessment, or a vendor recommendation.

Granola’s Security, Privacy & Data FAQs states that data is stored on AWS servers in the United States and that EU, UK, Canadian, Australian, and other regional residency options are not currently offered.

The same documentation also says that meeting notes are cached locally on the user’s device.

Granola’s separate security page states that notes are stored in a US-hosted AWS Virtual Private Cloud and backed up daily. It also explains that transcription providers and AI providers form part of the service workflow.

These statements are useful.

They identify the primary hosting environment.

They identify the absence of alternative regional residency options.

They also begin to show that the service involves more than one data location and more than one provider role.

They do not, by themselves, establish:

• where each transcription or AI provider processes data;

• which data types each provider receives;

• where backups and recovery copies are located;

• whether local caches follow the same deletion lifecycle;

• where support or incident-response access may occur;

• whether every plan and feature follows the same path;

• which geographic commitments appear in the customer’s contract.

The public profile helps locate the evidence.

The vendor’s own documentation remains the primary source.

Neither should be stretched beyond what it says.

What it does not prove

A data-residency statement does not automatically establish:

Data-type coverage. The statement may apply to notes, prompts, or outputs without covering uploaded files, transcripts, embeddings, metadata, logs, moderation records, support copies, or backups.

Processing location. Data can be stored in one country while being processed elsewhere by model, transcription, security, analytics, or infrastructure providers.

Secondary storage. Caches, indexes, vector stores, observability platforms, disaster-recovery environments, and support tools may follow different location rules.

Provider routing. The buyer may not know which model or service provider receives a request, whether fallback routing occurs, or whether provider selection changes by feature or region.

Human access location. A regional storage claim does not show where support, engineering, safety, or incident-response personnel may access customer data.

Exception paths. Abuse review, legal requests, security investigations, support cases, customer-enabled integrations, and service failures may alter the normal path.

Deletion scope. Deletion from the main application does not automatically establish deletion from local caches, logs, support systems, backups, or downstream providers.

Product and plan scope. A location statement may apply to one application, feature, plan, region, or deployment model without covering the exact service under review.

Contract scope. A public documentation page does not necessarily provide the same commitment as a DPA, order form, service agreement, regional addendum, or negotiated term.

None of these gaps proves that the vendor is handling data improperly.

They show that the location claim is narrower than the decision the buyer is trying to make.

Weak-answer pattern

This is the country-without-path pattern.

The vendor names a hosting region.

The buyer records:

“Data hosted in the United States.”

Or:

“EU residency available.”

But the review file does not identify:

• which data types are covered;

• where processing occurs;

• which providers receive the data;

• where logs and backups remain;

• where support access may occur;

• which exceptions change the path;

• which product and plan are covered;

• which contract term supports the boundary.

The answer may be accurate.

The geographic scope is still incomplete.

Evidence request

Do not ask only:

“Where is customer data hosted?”

Ask for the data-location map.

A stronger request is:

Please provide a product- and plan-specific data-location map for the intended workflow. For each relevant data type, identify where it is stored, where it is processed, which providers receive it, where backups and recovery copies are held, where human or support access may occur, which exceptions may alter the path, and which contractual source supports the stated regional boundary.

Then ask:

• Which data types are covered by the residency commitment?

• Does the stated region apply to both storage and processing?

• Which model, transcription, infrastructure, security, and analytics providers receive data?

• Can routing or fallback move processing to another provider or region?

• Where are logs, caches, indexes, backups, and recovery copies located?

• From which locations may support, engineering, safety, or incident personnel access the data?

• Do integrations or customer-enabled features change the path?

• Does the boundary differ by product, plan, configuration, or feature?

• Which DPA, order-form term, or regional addendum confirms the commitment?

The vendor does not need to disclose sensitive architecture.

It does need to establish a geographic boundary that the buyer can use.

Review note

A weak review note says:

“Customer data is hosted in the United States.”

A stronger review note says:

The vendor identifies a customer-data hosting region, but the available evidence does not establish the complete geographic scope of processing, provider routing, logs, caches, backups, support access, subprocessors, or exception paths. The response is not evidence-complete for a use case requiring verified in-region storage and processing.

That note does not say the residency statement is false.

It prevents a storage-location claim from being treated as proof of the complete data path.

Usage boundary

Until the geographic path is evidenced:

Limit use to data that does not require verified in-region storage and processing. Do not introduce regulated, highly confidential, customer-restricted, or contract-restricted data where uncertainty about processing locations, providers, backups, support access, or exception paths would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A residency claim can support that review.

It cannot replace the data path.

A country name is evidence.

It is not regional coverage.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRiskAssessment #DataResidency #AIGovernance

AI vendors often say:

“Audit logs are available.”

“Enterprise administrators can monitor activity.”

“Logs can be exported to your SIEM.”

These statements sound like evidence of accountability.

They may prove that some logging capability exists.

They do not prove that the events needed to reconstruct an AI workflow are actually recorded.

The review question is not only:

Does the vendor provide audit logs?

The better question is:

Which events are recorded, which events are missing, and can the buyer reconstruct what the AI system accessed, produced, changed, or exposed?

An audit log can exist while the decisive AI event remains invisible.

That is the evidence gap.

Claim

The vendor says:

“Enterprise customers have access to audit logs.”

Or:

“Key platform activity is logged.”

Or:

“Security and administrative events can be exported.”

These statements may establish that the product records certain identity, administrative, operational, or security events.

They do not establish that the actual AI activity is visible.

Why it sounds sufficient

Audit logs are familiar enterprise controls.

They suggest that administrators can monitor activity, investigate incidents, identify configuration changes, and retain evidence for internal review.

A vendor questionnaire may therefore ask whether audit logs exist and treat a “yes” answer as complete.

But conventional SaaS logs often focus on:

• login and logout;

• account creation;

• role and permission changes;

• workspace settings;

• administrative actions;

• security alerts.

An AI workflow may create additional events:

• prompts and model invocations;

• uploaded files and connector access;

• retrieval of documents or records;

• model-provider routing or fallback use;

• tool or agent execution;

• generated outputs and external sharing;

• human review or support access;

• retention and policy-setting changes;

• blocked, failed, overridden, or retried actions.

If those events sit outside the logging scope, the buyer may have an enterprise log without having an AI activity record.

The feature may be real.

The incident-reconstruction capability may still be incomplete.

What it actually proves

A current, product- and plan-specific audit-log statement may establish that:

• an audit-logging mechanism exists;

• certain identity or administrative events are recorded;

• administrators or vendor personnel can review selected activity;

• logs may be exportable through an API or SIEM integration;

• the vendor maintains operational records for security or troubleshooting.

These are meaningful evidence points.

They establish that logging exists.

They do not identify the complete event taxonomy, field coverage, retention period, customer-access model, export capability, or known exclusions.

Availability is not scope.

A public evidence example

The CodeYourCompliance public evidence profile for Decagon records an audit-logging claim among the public enterprise-readiness evidence observed for Decagon’s AI customer-support platform.

The profile is an evidence index.

It is not verification that customers can access the logs or that the logging scope is sufficient for incident reconstruction.

Decagon’s security page states that tamper-protected logs capture key events ranging from logins to unusual activity. The same page says access to those logs is limited to senior engineering leadership.

A separate security and compliance page states that Decagon maintains audit logs of activity, errors, and warnings on production systems.

These statements provide useful evidence that internal logging mechanisms exist.

They do not establish that enterprise customers receive a customer-facing audit trail.

The public materials do not identify:

• the complete event taxonomy;

• which AI-specific actions are recorded;

• whether customer administrators can access or export the records;

• how long the logs are retained;

• whether prompts, connector retrievals, model routes, tool calls, human access, policy changes, exports, and failed actions are included;

• which important events are explicitly excluded.

The evidence therefore supports a narrow conclusion:

The vendor publicly describes internal audit-logging mechanisms.

It does not establish the event-level visibility available to the buyer for the intended workflow.

What it does not prove

An audit-log claim does not automatically establish:

AI event coverage. The log may not record model invocation, prompt submission, retrieved context, file use, generated output, tool execution, or agent action.

Data-access visibility. The buyer may not be able to see which connector, repository, document, database, or customer record was accessed.

Routing visibility. The log may not identify which model provider, region, deployment, fallback route, or integration handled the request.

Human and administrative access. Support access, abuse review, human evaluation, retention changes, permission changes, connector changes, and model-selection changes may not be visible.

Actor attribution. The record may not distinguish users, administrators, service accounts, vendor personnel, contractors, and automated agents.

Outcome visibility. The event may not show whether an action succeeded, failed, was blocked, was retried, or was overridden.

Evidence usability. The buyer may not know whether timestamps are consistent, fields are documented, records are retained long enough, logs can be exported, or administrators can alter or delete them.

A vendor may have strong authentication and infrastructure logs while leaving the customer-relevant AI workflow largely unobservable.

None of these gaps proves that the events are absent.

They show that the available evidence does not establish whether the buyer can reconstruct them.

Weak-answer pattern

This is the feature-without-event-scope pattern.

The vendor confirms that audit logs exist.

The buyer records:

“Audit logs available.”

But the review file does not identify:

• which events are included;

• which AI-specific events are excluded;

• who can access the records;

• how long they are retained;

• whether customers can export them;

• whether the records support incident reconstruction.

The answer describes the feature.

It does not establish the evidence boundary.

Evidence request

Do not ask only:

“Do you provide audit logs?”

Ask for the event taxonomy.

A stronger request is:

Please provide the audit-event taxonomy for the exact product and plan under review, including AI-specific events, administrative events, human-access events, tool execution, data access, routing, outcome status, and known exclusions.

For each event type, identify:

• actor and actor type;

• timestamp;

• action;

• target object;

• success, failure, block, retry, or override result;

• feature, integration, connector, model, or route involved;

• customer-access model;

• log-retention period;

• export method;

• documentation source.

Then ask:

• Are prompts, model invocations, and generated outputs represented as events?

• Are connector access, retrieved sources, file actions, and tool execution visible?

• Are model routing, fallback use, and agent actions recorded?

• Are support access, human review, and administrative changes logged?

• Can customers export the records, and how long are they retained?

• Which relevant events are explicitly excluded?

The vendor does not need to expose model internals.

It does need to show whether the buyer can reconstruct the events that matter for the intended use case.

Review note

A weak review note says:

“Audit logs available.”

A stronger review note says:

The vendor publicly describes audit-logging mechanisms, but the current evidence does not establish customer access, export capability, retention, or coverage of AI-specific events such as model invocation, connector access, routing, tool execution, human review, output sharing, and policy changes. The response is not evidence-complete for incident reconstruction or accountability in the intended workflow.

That note does not say the vendor lacks logging.

It says the buyer does not yet know whether the available logs support the decision being made.

Usage boundary

The required log scope depends on what the AI system is allowed to do.

A drafting assistant used with low-sensitivity internal content may require less event detail than an agent that accesses customer records, executes tools, changes systems, or produces external communications.

Until the relevant event scope is evidenced:

Limit use to workflows where missing audit events would not prevent incident investigation, accountability, or reversal. Do not rely on the vendor’s audit logs for high-impact actions, sensitive data access, autonomous execution, system changes, or external decisions until event coverage, customer access, retention, export, and known exclusions are confirmed.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

Audit-log availability can support that review.

It cannot replace event-level scope.

A log that omits the decisive event cannot support the conclusion the buyer expects from it.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRiskAssessment #AuditLogs #AIGovernance

2026-07-29: PASS
2026-07-30: FAIL

The first question is whether the host changed or the test changed. The record cannot answer it.

The package inventory may have moved, but the baseline, freshness rule, policy or evaluator may also have changed. The same evidence can produce a different result when it is judged under a different evaluation context.

With only PASS and FAIL, there is no way to separate a change in the system from a change in the decision conditions. The conclusions were retained. The basis for them was not.

That is why a stored control result is not replayable evidence. An earlier note asked whether audit evidence can survive replay; the harder implementation question is what must be retained for that replay to mean anything.

What must survive the original decision

Replayable evidence is not the same as retained documentation. Replay does not mean running the same scanner again.

A new scan observes the system as it exists now. Replay reconstructs a previous decision from the artifacts that existed when that decision was made.

For that reconstruction to be credible, the retained package must include more than the observation. It must also preserve the schema that admitted the evidence, the context that supplied the threshold or baseline, the policy artifact, the selected decision engine, the implementation version and an integrity digest binding the assessment to the original evidence bytes.

Each artifact answers a different review question.

The evidence shows what the collector observed. The schema shows what input was considered admissible. The context records the rule parameters used at that time. The policy contains the decision logic. Runtime metadata identifies the execution path. The digest shows whether the retained evidence has changed since evaluation.

If any of these elements is missing, part of the decision must be inferred rather than replayed.

A timestamped result may still be useful operationally, but it is only a historical claim unless the decision path can be reconstructed.

The collector should not decide compliance

Consider a read-only collector operating within an explicit audit boundary. It returns the following package inventory:

{
  "packages": [
    {"name": "openssl", "version": "3.0.13"},
    {"name": "python3", "version": "3.12.3"},
    {"name": "curl", "version": "8.5.0"}
  ]
}

This tells us what the collector observed. It does not tell us whether the host complies with a package baseline.

That decision depends on a separate context:

{
  "required_packages": [
    {"name": "openssl", "version": "3.0.13"},
    {"name": "python3", "version": "3.12.3"},
    {"name": "curl", "version": "8.5.0"}
  ],
  "unexpected_package_mode": "ignore"
}

If the required OpenSSL version changes, the same inventory may fail. If unexpected packages are changed from ignore to fail, an additional package may alter the result even though nothing on the host has changed.

The observation and the assessment therefore belong to different layers.

Package names and versions come from collection. Missing packages, mismatched versions and unexpected packages are derived by comparison. Pass or fail is produced only after policy evaluation.

Writing the control result back into the evidence object hides those transitions. The final document becomes easier to read, but harder to inspect. A reviewer can see the conclusion but cannot tell which facts came from the host, which were derived by the replay engine and which depended on policy choices.

Invalid, stale and failed are different states

A control should fail only after admissible evidence reaches the policy and the required condition is not met.

If the evidence does not satisfy its schema, the policy should not evaluate it. A missing timestamp, malformed package entry or incorrect field type is a problem with the evidence path, not proof that the host failed the control.

The assessment should record that distinction:

evidence_status = invalid_evidence
control_status = unknown
decision_executed = false

Evidence can remain intact and still become too stale to support a current conclusion. In this control, stale evidence creates a different problem from either invalid evidence or policy failure. A package inventory may be structurally valid and still be too old for the decision being attempted.

Suppose the context requires evidence collected within the previous 24 hours. An inventory from three days ago cannot support a current package-baseline decision. It does not prove that the host is non-compliant. It proves that the available observation is no longer fresh enough to answer the question.

evidence_status = stale_evidence
control_status = unknown
decision_executed = false

These distinctions are not cosmetic. They imply different actions.

A failed control may require investigation of the host or baseline. Stale evidence requires recollection. Invalid evidence requires examination of the collector, schema or processing pipeline.

A dashboard that reduces all three conditions to red has removed information that operations and audit both need.

Collection and policy execution are separate boundaries

The earlier implementation note moved from evidence principles to an executable control. The public reference artifacts now apply that structure through a fixed replay sequence:

evidence schema gate

-> context schema gate

-> integrity gate

-> freshness gate

-> derived facts

-> policy evaluation

-> assessment output

For the package-baseline example, the comparison stage derives three groups of facts:

• missing_packages

• version_mismatches

• unexpected_packages

The policy then decides how those facts affect the control result.

A missing required package causes failure. A version mismatch causes failure. An unexpected package may be ignored or treated as a violation depending on the explicit context.

The collector does not need to know any of this. Its job is to report the inventory accurately and preserve its collection metadata. Policy belongs later in the pipeline, where its input and execution can be inspected independently.

The same boundary applies to the decision engine.

A built-in evaluator can implement the same decision semantics as a Rego policy, but it does not execute the Rego artifact. That difference must appear in the assessment:

builtin:
decision_executed = true
policy_artifact.executed = false

opa:
decision_executed = true
policy_artifact.executed = true

Without this metadata, a result may appear to have been produced by a policy artifact that was never run. The conclusion could be correct while its provenance is false.

Replay should stop before remediation

The package-baseline control can identify a missing package, an incorrect version or an unexpected installation. It does not install, remove or upgrade software.

That is a deliberate boundary.

Remediation requires broader permissions and produces a different class of evidence: authorization records, execution logs, change results and post-remediation observations. Combining those actions with collection makes it harder to establish what state existed before the tool intervened.

A read-only collector preserves the condition it was asked to observe. A separate remediation workflow may act on the assessment later, but it should produce its own evidence package.

Collection establishes what was seen. Correction changes what will be seen next.

Public reference artifacts

The replay structure described here is implemented in the public codeyourcompliance/evidence-validation-pipeline repository.

The current examples are:

• examples/replayable-tls-control

• examples/os-package-baseline

The TLS example applies the pipeline to certificate expiry. The OS package example applies it to required packages, version mismatches and unexpected software.

Both packages retain the evidence, validation schemas, evaluation context, integrity information, policy artifact and execution metadata needed to inspect how an assessment result was produced. Their tests also show how the pipeline behaves when evidence is stale, malformed or modified after hashing.

These examples do not prove that the underlying systems are compliant. They show that a control decision can be preserved in a form that is more inspectable than a stored PASS or FAIL.

They also stop before remediation. Installation, removal, upgrade and other corrective actions belong to a separate workflow with separate permissions and evidence.

Replayability does not establish that a decision was correct. It establishes whether the basis of that decision can still be examined.

Origin and scope

CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• From Evidence Principles to an Executable Control

• Can Your Audit Evidence Survive Replay?

• Why Audit Evidence Goes Stale

• Documents are not system state.

• Screenshots are supporting artifacts, not proof objects.

• Evidence should be timestamped, source-bound, integrity-checked, and replayable.

• Audit evidence must remain current enough to support a conclusion.

• Automatically generated reports are not accountable audit conclusions.

These ideas have shaped almost every article I’ve written.

But they also raised an uncomfortable question.

If these principles only exist in articles, can they really be tested, challenged, or reproduced?

That question has led to the next stage of the project.

Instead of treating articles as the primary deliverable, I will begin treating each implementation as the primary object. Articles become explanations of that implementation rather than the implementation itself.

The goal is to turn individual evidence principles into small, replayable engineering experiments that anyone can inspect, critique, and improve.

The first reference experiment is intentionally narrow.

TLS endpoint
↓
read-only collection
↓
raw evidence
↓
parsed facts
↓
policy evaluation
↓
assessment result
↓
OSCAL representation

The objective is not to build a complete compliance platform.

The objective is to answer one implementation question:

Can a narrow, replayable implementation preserve the distinction between collection, evidence, policy evaluation, and accountable assessment?

Several boundaries are equally important.

This experiment does not claim that a successful assessment proves overall compliance, nor does it attempt to automate professional judgment.

The purpose is much smaller—and, I believe, much more useful.

If a single evidence principle can be implemented, replayed, inspected, and shown to fail under controlled conditions, it becomes more than an idea. It becomes something other people can verify, criticize, and build upon.

That is the direction CodeYourCompliance is taking next.

Not more articles.

More executable knowledge.

Origin

CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This project does not provide legal, regulatory, audit, certification, or compliance advice.

Project Direction

Future articles will increasingly accompany replayable implementation packages rather than stand alone as methodology notes.

Related Reading

• Compliance Automation Starts at Evidence

• Can Your Audit Evidence Survive Replay?

• What a MAS TRM Checklist Cannot Prove

• A Screenshot Is a Supporting Artifact, Not a Proof Object

• A Generated Report Is Not an Accountable Audit Conclusion

AI vendors often describe retention as a setting:

“Zero data retention is available.”

“Enterprise customers can configure retention.”

“Prompts are deleted after 30 days.”

These statements sound operational.

They suggest that the buyer controls the data lifecycle.

But a retention setting is not automatically a retention control.

The review question is not only:

What retention period does the vendor advertise?

The better question is:

What data lifecycle does the available evidence establish for this product, plan, feature, configuration, region, and use case?

A setting may control one data object in one part of the service.

It may not cover files, metadata, logs, embeddings, support copies, caches, backups, or exception paths.

That is the evidence gap.

Claim

The vendor says:

“Enterprise customers can configure prompt retention.”

Or:

“Zero data retention is enabled by default.”

Or:

“Customer content is deleted after 30 days.”

These statements may establish that a retention feature or policy exists for a defined category of data.

They do not establish the complete lifecycle of the buyer’s data.

Why it sounds sufficient

Retention is often reduced to one number:

• zero days;

• thirty days;

• customer-configurable;

• deleted on request.

That makes the answer easy to enter into a questionnaire and mark complete.

But an AI workflow rarely creates only one data object.

A single interaction may create:

• prompts and outputs;

• uploaded files;

• embeddings or indexes;

• metadata and usage records;

• security or moderation events;

• logs and diagnostic records;

• support artifacts;

• cached copies;

• backups.

The vendor may use “retention” to describe visible conversation history.

The buyer may assume that the same setting applies to every object created by the workflow.

It may not.

A prompt can disappear from the user interface while related metadata remains in an operational log.

A file can be removed from active storage while a backup copy remains until rotation.

A zero-retention inference endpoint can coexist with another feature that stores conversations by default.

The setting may be real.

The assumption that it controls the entire lifecycle is the problem.

What it actually proves

A current, product-specific retention statement may establish that:

• a retention feature exists;

• a defined data type follows a stated retention period;

• administrators can shorten or disable certain storage;

• a product or plan provides additional lifecycle controls;

• the vendor has documented a customer-facing retention option.

These are useful evidence points.

They establish something about a defined setting or data category.

They do not automatically establish the lifecycle of every object, storage layer, feature, or exception path involved in the buyer’s workflow.

A public evidence example

The CodeYourCompliance public evidence profile for Fireworks AI records zero data retention as a public evidence signal and points to the vendor’s data-handling documentation.

The profile is an evidence index.

It is not implementation verification for a particular customer account, endpoint, configuration, or request.

Fireworks AI’s data-retention documentation shows why scope matters.

For open-model inference, the documentation states that prompt and generation data is not logged or stored persistently by default without explicit user opt-in. It also states that service metadata, such as token counts, is logged.

The same documentation explains that certain advanced features allow customers to opt into prompt and generation logging.

Separately, the Fireworks Responses API follows a different lifecycle. When store=True, which is the documented default, complete conversation data is retained for 30 days. This may include user prompts, model responses, and tools called by the model.

Storage can be disabled by setting store=False.

These statements are not necessarily contradictory.

They describe different services, endpoints, data types, features, and configurations.

For one path, prompt and generation data may not be stored persistently.

For another path, conversation data may be stored by default for a defined period.

Metadata may still be logged.

Advanced features may introduce opt-in content storage.

That is exactly why “zero retention” cannot remain a vendor-level label.

The buyer must determine which lifecycle applies to the exact workflow being reviewed.

What it does not prove

A configurable retention statement does not automatically establish:

Data-type coverage. The setting may cover prompts and outputs without covering files, embeddings, metadata, logs, moderation records, support artifacts, or backups.

Storage-layer coverage. Deletion from the primary application does not establish deletion from caches, indexes, vector stores, observability platforms, support systems, or recovery environments.

Deletion timing. “Deleted” may mean removed immediately, queued for deletion, removed from active systems later, or retained in backups until scheduled rotation.

Exception paths. Data may follow different retention rules for security investigations, abuse monitoring, customer support, legal obligations, incident response, or customer-enabled features.

Configuration scope. The lifecycle may depend on an endpoint parameter, administrator setting, feature toggle, workspace default, or customer opt-in.

Administrative enforcement. A setting does not show whether changes are logged, whether access is restricted, whether the customer can verify enforcement, or whether new features inherit the same policy.

Product and plan scope. API products, hosted applications, enterprise plans, individual plans, integrations, beta features, and model routes may follow different retention rules.

Contract scope. Public documentation does not necessarily provide the same commitment as a DPA, order form, service agreement, product addendum, or customer-specific term.

None of these gaps proves that the vendor retains data improperly.

They show that a retention statement may be narrower than the lifecycle the buyer needs to understand.

Weak-answer pattern

This is the setting-without-lifecycle pattern.

The vendor provides a retention period or configuration option.

The buyer records:

“Zero retention available.”

Or:

“Prompts deleted after 30 days.”

But the review file does not identify:

• which data types are covered;

• which endpoint or feature the setting controls;

• whether it applies by default;

• which storage layers are included;

• whether metadata or logs remain;

• which exceptions apply;

• how deletion is verified;

• which contractual source supports the claim.

The vendor has described a setting.

The buyer has not yet established the control boundary.

Evidence request

Do not ask only:

“What is your prompt-retention period?”

Ask for the lifecycle.

A stronger request is:

Please provide a product- and plan-specific retention matrix for the AI workflow under review. For prompts, outputs, uploaded files, embeddings, metadata, logs, moderation events, support artifacts, caches, indexes, and backups, identify the default and configurable retention period, storage layer, deletion timing, exception paths, human-access boundary, customer controls, audit evidence, and supporting contractual source.

Then ask:

• Does the setting apply by default or require configuration?

• Which endpoints, features, and data types are covered?

• Does a setting change apply to existing data or only newly created data?

• Which metadata, security records, or operational logs remain after content deletion?

• How are cached, indexed, backed-up, or support-system copies handled?

• Can the vendor override the setting for support, safety, abuse, incident, or legal purposes?

• Are configuration changes recorded in audit logs?

• Do new features and integrations inherit the same retention policy?

• Which DPA, order-form term, or product addendum confirms the lifecycle?

The vendor does not need to disclose sensitive architecture.

It does need to establish an operationally usable boundary.

Review note

A weak review note says:

“Prompt retention is configurable.”

A stronger review note says:

The vendor identifies a configurable retention feature, but the available evidence does not establish the complete data lifecycle for the intended workflow. Coverage of files, embeddings, metadata, logs, support artifacts, caches, indexes, backups, configuration changes, and exception paths remains unclear. The response is not evidence-complete for data requiring verified deletion, short-lived processing, or customer-controlled retention.

That note does not say the vendor’s statement is false.

It prevents a setting from being treated as proof of the entire lifecycle.

Usage boundary

Until the complete retention lifecycle is evidenced:

Limit use to low-sensitivity internal or test data. Do not introduce customer-confidential data, regulated data, source code, employee records, privileged business material, or other data requiring verified deletion where uncertainty about storage layers, logs, metadata, backups, exception paths, or configuration would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A retention setting can support that review.

It cannot remain a single number inside it.

A setting is part of the evidence.

It is not the whole control.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorAssessment #AIGovernance #ThirdPartyRisk #Privacy #VendorRisk

Authentication succeeds.

Configuration data appears in the dashboard.

The control status turns green.

Then the reviewer asks:

Which source produced this value?

Which collector version retrieved it?

When was it observed?

Was the full target scope reached?

Was the data changed during normalization?

Which policy evaluated it?

The integration moved the data.

It did not preserve the answers.

That is the boundary.

What an Integration Proves

A working integration may prove that:

• credentials were accepted

• an endpoint was reachable

• a payload was returned

• field mapping completed

• data reached the destination

That is useful operational evidence.

It does not automatically prove that the transported data is valid audit evidence.

The integration may still omit:

• source context

• expected collection scope

• failed targets

• collector identity and version

• collection timestamp

• transformation history

• integrity status

• evidence freshness

• applicable policy version

Compliance evidence automation is not data movement.

It means collecting evidence from source systems, preserving provenance, validating integrity, evaluating policy, and producing a reviewable evidence package.

Missing Data Must Survive the Integration

Suppose a collector is expected to inspect 100 systems.

It reaches 94.

The integration writes 94 successful records into the compliance platform.

The dashboard reports that all received records passed.

Technically, the integration worked.

The evidence package is incomplete.

The correct state is:

expected targets: 100

observed targets: 94

failed collection: 6

It is not:

evaluated environment: compliant

The six missing systems did not fail the control.

They did not pass it either.

They were not observed.

If that distinction disappears during transport, the evidence has already lost part of its value.

What an Evidence Contract Preserves

An evidence contract defines the minimum structure that must survive collection, transport, normalization, evaluation, and reporting.

evidence_contract:
  source_system:
  collector:
  collector_version:
  collection_method:
  timestamp_utc:
  evidence_type:
  evidence_layer:
  integrity_hash:
  validation_status:
  policy_input_ref:

Each field answers a different audit question.

source_system identifies where the evidence originated.

collector and collector_version identify what process observed the source.

collection_method distinguishes read-only observation from collection that may have changed the assessed system.

timestamp_utc records when the state was observed.

evidence_type defines what kind of object was collected.

evidence_layer separates raw evidence, normalized facts, derived facts, policy results, and audit narratives.

integrity_hash supports detection of post-collection mutation.

validation_status preserves whether the evidence is valid, stale, missing, tampered, unknown, or not machine-verifiable.

policy_input_ref links the evidence to the rule evaluation that consumed it.

Without these fields, the destination may contain data.

It does not yet contain a replayable evidence object.

One Green Status Is Not Enough

A common implementation error is using one status field for the entire pipeline:

{
  "status": "success"
}

What succeeded?

The API request?

The collection?

The schema validation?

The integrity check?

The policy evaluation?

The control?

These outcomes should remain separate.

transport_status: success
collection_status: partial
schema_status: valid
integrity_status: verified
freshness_status: stale
policy_status: not_evaluated
control_status: unknown

A single green status is easy to display.

It is difficult to audit.

Normalization Changes Meaning

Integrations often rename fields, convert timestamps, flatten objects, infer Boolean values, and discard unsupported data.

These transformations may be necessary.

They are not neutral.

Suppose the source returns:

{
  "certificate_not_after": "2026-08-01T00:00:00Z"
}

The integration writes:

{
  "certificate_valid": true
}

The second value is not raw evidence.

It is a derived fact.

The system should preserve the source value, transformation rule, evaluation time, and derived result.

Otherwise the integration replaces evidence with an unexplained conclusion.

OPA Cannot Repair Weak Evidence

OPA can evaluate a defined input against a defined policy.

It cannot determine whether the input was collected from the correct source, whether the target scope was complete, or whether the evidence was modified before evaluation.

A reproducible rule result can still be based on weak evidence.

The policy engine is not the point.

The audit problem starts earlier.

The evidence contract defines what the evaluator is allowed to treat as policy input.

Different Failures Need Different Outcomes

The pipeline should distinguish

An API timeout is not automatically non-compliance.

The defensible chain is:

Control failure should be recorded only when valid evidence was evaluated and the required condition was not met.

Failure states must survive the integration.

Technical companion

An evidence contract boundary note and two synthetic integration examples are available in the CodeYourCompliance evidence-validation-pipeline repository.

Evidence contract boundary:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/docs/evidence_contract.md

Integration without an evidence contract:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/integration_without_evidence_contract.json

Integration with an evidence contract:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/integration_with_evidence_contract.json

These examples do not implement a complete integration governance or evidence integrity model.

They show one boundary:

Successful transport does not establish evidence validity.

Boundary

This material is for evidence structuring, integration design review, and technical review preparation.

It does not provide legal, regulatory, audit, certification, compliance, procurement, or implementation advice.

The goal is not to prove MAS TRM compliance.

The goal is to make the transport-versus-evidence boundary visible before synchronized data becomes accepted audit evidence.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Related reading

Why Audit Evidence Goes Stale

Can Your Audit Evidence Survive Replay?

What a MAS TRM Checklist Cannot Prove

A Screenshot Is a Supporting Artifact, Not a Proof Object

A Generated Report Is Not an Accountable Audit Conclusion

A subprocessor list is useful evidence.

It is not the actual AI data path.

An AI vendor may publish a list of third-party service providers. That list may show which subprocessors the vendor is permitted to use.

But for a buyer, the review question is narrower:

Which provider touches which data, for which product feature, in which region, under which contract boundary, for which use case, and for how long?

If that is not answered, the subprocessor list remains source material.

It is not evidence-complete for AI vendor review.

Claim

The vendor says:

“We maintain a subprocessor list.”

Or:

“Our current subprocessors are listed here.”

Or:

“We only share customer data with approved subprocessors.”

Or:

“Customers will be notified of subprocessor changes.”

These statements may support a baseline third-party disclosure process.

They do not automatically show how data moves through the actual AI workflow.

Why it sounds sufficient

A subprocessor list feels concrete.

It usually includes company names, service descriptions, locations, and sometimes update dates or notification terms.

For a standard SaaS review, this may support a basic third-party processing check. The buyer can see that cloud infrastructure, analytics, support, security tooling, logging, email delivery, or AI infrastructure providers may be involved.

That matters.

But AI vendor review usually needs more than a list.

A buyer may be sending prompts, files, meeting audio, transcripts, source code, customer records, support tickets, embeddings, metadata, logs, or generated outputs into a specific product workflow.

A static subprocessor list may not show which of those data types each third party touches.

It may not show whether a model provider processes prompts and outputs. It may not show whether transcription, embedding, retrieval, safety filtering, support access, logging, analytics, or abuse monitoring follow different paths.

The list names possible processors.

It does not explain the operating path.

What it actually proves

A subprocessor list may show that the vendor discloses third parties, identifies their general functions, maintains an update process, and may notify customers of changes.

These are useful evidence points.

They establish disclosure practices.

They do not establish the buyer’s actual AI workflow.

A public evidence example

The CodeYourCompliance public evidence profile for Vercel records several public evidence surfaces observed as of 27 June 2026, including the Trust Center, DPA, subprocessor disclosures, AI data-use terms, and the separate v0 Enterprise Addendum.

The profile is an evidence index. It is not verification of an operating data path.

For v0, Vercel’s public Trust Center materials identify certain providers and describe functions such as model inference, AI services, monitoring, and analytics.

Separately, Vercel’s AI Gateway documentation describes provider ordering and fallback behaviour for AI Gateway.

These are different product surfaces.

They should not be combined into one product-level conclusion.

Together, they illustrate the evidence problem: a subprocessor disclosure may identify possible participants, while product documentation may describe possible routing behaviour.

Neither source alone shows which provider handled a particular request, what data was sent, whether fallback occurred, what the provider retained, or whether the same path applied to the buyer’s product, plan, configuration, and region.

The list identifies possible participants.

The product documentation describes possible routing.

The buyer still needs the workflow-level map.

What it does not prove

A subprocessor list does not automatically establish:

Product and feature use. A provider listed at company level may not participate in the exact product, feature, plan, or configuration under review.

Data-type coverage. The list may not show whether a provider receives prompts, outputs, files, transcripts, embeddings, metadata, logs, support records, or safety events.

Routing and fallback. It may not show which provider handles a request, whether routing changes by region, or whether another provider is used after failure, timeout, or capacity constraints.

Retention and access. It may not explain what each provider stores, how long it is retained, who can access it, or how deletion is propagated.

Contract scope. It may not show whether the relevant processing path is covered by the customer’s DPA, order form, product addendum, objection rights, or regional commitments.

The buyer may therefore record:

“Subprocessor list reviewed.”

But the review file still does not establish the actual AI data path.

Weak-answer pattern

This is the list-without-routing pattern.

The vendor provides a subprocessor list, and the buyer records it as if the data path were understood.

But the list does not show which provider handles prompts, embeddings, logs, support records, or fallback traffic for the specific workflow.

Without that mapping, the list is a directory.

It is not the data path.

Evidence request

Do not ask only:

“Can you provide your subprocessor list?”

Ask for feature-level routing.

A stronger request is:

Please map the subprocessors and model providers involved in the specific product, plan, region, and AI workflow under review, including which data types each provider receives, processes, stores, logs, or can access.

Then ask:

• Which providers are used for the exact product, plan, feature, configuration, and region?

• Which data types does each provider receive, process, store, log, or access?

• Can model routing, fallback, support, monitoring, or safety workflows change the provider path?

• What retention, deletion, and human-access terms apply at each provider?

• Which DPA, addendum, order-form term, or customer control confirms the boundary?

That is how a subprocessor list becomes reviewable.

Review note

A weak review note says:

Subprocessor list available.

A stronger review note says:

The vendor provides a subprocessor list, but the list does not by itself establish the actual AI data path for the intended workflow. The buyer should confirm which subprocessors and model providers process each relevant data type, which product features use them, whether routing differs by plan, region, or configuration, whether support and monitoring systems receive customer content, and which contract terms confirm the processing boundary.

That note does not say the vendor is unsafe.

It says the subprocessor evidence has not yet been mapped to the workflow.

Usage boundary

Until provider routing is mapped, usage should remain bounded.

Limit use to low-sensitivity internal data. Do not introduce customer-confidential data, regulated data, source code, employee records, or privileged business material where uncertainty about provider routing, retention, support access, or regional handling would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A subprocessor list can support the review.

It cannot replace the actual AI data path.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply to request the sanitized sample evidence gap memo.

The export is real.

The timestamp is real.

The file hash still verifies.

But the system changed after collection.

The evidence is intact.

The conclusion is not.

That is the freshness problem.

A verified hash does not make stale evidence current.

Evidence can be true and stale at the same time

Compliance teams often treat evidence as a static artifact.

A screenshot was captured.

A report was exported.

A log bundle was saved.

A file hash was recorded.

That may prove something useful.

It does not prove the evidence still supports the conclusion being made today.

Integrity asks whether the evidence changed after collection.

Freshness asks whether the evidence is still recent enough to support the current claim.

Those are different questions.

A stale evidence object can still pass integrity verification.

That is the trap.

Freshness is an evidence property

Compliance evidence freshness is the condition that an evidence object remains recent enough, within its expected validity window, to support the compliance conclusion being drawn from it.

Freshness is not decoration.

It is not a report date.

It is not the date the auditor opened the folder.

It belongs inside the evidence object.

A minimum freshness record should preserve:

evidence_freshness:
  collected_at:
  valid_until:
  freshness_window_seconds:
  freshness_status: fresh | stale | expired | unknown

The collection timestamp tells us when the evidence was observed.

The validity window tells us how long that evidence can reasonably support the claim.

The freshness status tells us whether downstream evaluation should proceed.

Without this, the report can quietly reuse old evidence as if it were still current.

Integrity is not freshness

A hash can prove that an evidence object has not changed.

It cannot prove that the system has not changed.

A TLS certificate export can be hashed.

A firewall rule export can be hashed.

An access review file can be hashed.

A cloud configuration snapshot can be hashed.

If the source system changes after collection, the hash still verifies.

The evidence object is stable.

The world it describes may not be.

This is the boundary:

Both are needed.

One does not replace the other.

Why stale evidence should not become pass or fail

A stale evidence object should not silently produce a clean control result.

It should not be converted into pass.

It should not be converted into fail.

The correct outcome is a freshness classification.

For example:

verified evidence + expired freshness window = stale_evidence

That is not the same as non-compliance.

It means the evidence can no longer safely support the current conclusion.

The control may be working.

The control may have failed.

The evidence is no longer strong enough to decide.

That distinction matters.

Automation should preserve it.

The evidence quality gate

Policy evaluation should not begin with the control.

It should begin with the evidence.

The pipeline should first ask:

Is the evidence intact?

Is the evidence source clear?

Is the collector identified?

Is the collection time preserved?

Is the evidence fresh enough?

Only then should the system evaluate the control expectation.

A simple evidence quality gate looks like this:

evidence object

-> integrity verification

-> freshness classification

-> derived facts

-> policy evaluation

-> audit narrative

If freshness fails, downstream evaluation should stop or explicitly carry the stale evidence status.

The report should not hide the freshness failure.

A stale evidence example

In the CodeYourCompliance evidence validation pipeline, the sample stale evidence object uses a TLS certificate evidence record.

The evidence was collected on July 1.

It was evaluated on July 7.

Its declared freshness window was 24 hours.

The integrity status is still verified.

The freshness status is stale.

The policy result is not pass or fail.

It is:

{
  "input_status": "verified_but_stale_evidence",
  "result": "stale_evidence"
}

That is the correct boundary.

The evidence object survived integrity verification.

It did not survive freshness evaluation.

What the audit narrative should say

A weak audit narrative says:

TLS evidence reviewed. No issue noted.

A better audit narrative says:

That is not legal interpretation.

That is evidence discipline.

The narrative should explain what the evidence can still support.

It should also explain what it can no longer support.

MAS TRM-inspired framing

MAS TRM-inspired does not mean MAS TRM prescribes a freshness field.

It does not mean a specific JSON structure, Rego policy, hash algorithm, or evidence pipeline is required.

The framing is narrower.

If a compliance conclusion depends on technical evidence, the evidence should be recent enough to support that conclusion.

That is an engineering interpretation.

It separates the control claim from the proof material behind it.

Freshness changes the report boundary

Reports often collapse time.

They present collected evidence, reviewer notes, control statements, and conclusions inside one document.

That can make old evidence look current.

A freshness field forces the report to show the time boundary.

It asks:

When was this evidence collected?

How long was it valid for this claim?

When was the conclusion made?

Did the evidence remain fresh at conclusion time?

If the answer is no, the report should say so.

The evidence may still be useful background.

It should not be treated as current proof.

Technical companion

A stale evidence example and a narrow OPA/Rego freshness policy demo are available in the CodeYourCompliance evidence-validation-pipeline repository.

Stale evidence example:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/stale_evidence.json

Evidence freshness policy demo:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/policies/demo/evidence_freshness.rego

These examples do not implement a complete evidence integrity model.

They show one boundary:

verified evidence can still be stale evidence

Boundary

This material is for evidence structuring and technical review preparation.

It does not provide legal, regulatory, audit, certification, compliance, procurement, or implementation advice.

The goal is not to prove MAS TRM compliance.

The goal is to make the evidence freshness boundary visible before stale evidence becomes a clean report conclusion.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Related reading

Can Your Audit Evidence Survive Replay?

What a MAS TRM Checklist Cannot Prove

A Screenshot Is a Supporting Artifact, Not a Proof Object

A Generated Report Is Not an Accountable Audit Conclusion

Tool Approval Is Not Workflow Proof

Human review can be a safety control.

It can also be a data exposure path.

Both can be true.

An AI vendor may say human review is used for safety, abuse prevention, support, troubleshooting, quality assurance, or model evaluation.

That may sound responsible.

But for a buyer, the review question is narrower:

Who can see customer data, under what condition, for what purpose, for how long, in which system, under which contract boundary, and with what customer control?

If that is not answered, “human review” remains too vague for AI vendor assessment.

Claim

The vendor says:

“Human review may be used for safety.”

Or:

“Authorized personnel may access customer data for support and troubleshooting.”

Or:

“Customer content may be reviewed in limited circumstances.”

These statements may describe legitimate operational controls.

They do not automatically establish whether the exposure is bounded.

Why it sounds sufficient

Human review sounds responsible.

It suggests that the vendor is not relying blindly on automation. Human involvement may help investigate abuse, resolve support issues, troubleshoot product failures, review security incidents, or assess flagged content.

That matters.

But in an AI vendor review, human review is not only a control.

It is also a path through which customer data may become visible to people outside the buyer’s organization.

A general statement such as “limited human review” does not show which content can be accessed, who can access it, what triggers access, or where the reviewed information goes.

The buyer therefore needs more than a description.

The buyer needs a boundary.

What it actually proves

A human-review statement may establish that the vendor has a process through which authorized personnel can inspect certain content or operational records under defined circumstances.

It may support limited conclusions such as:

• the vendor investigates safety or abuse events;

• support personnel can troubleshoot customer issues;

• internal personnel may respond to security incidents;

• the vendor maintains operational review workflows.

These are useful evidence points.

They establish that a review process exists.

They do not establish its complete access boundary.

A public evidence example

The CodeYourCompliance public evidence profile for OpenAI records several public evidence surfaces, including the Trust Portal, security and privacy material, Services Agreement, DPA, enterprise privacy controls, and administrative-control references.

The profile is an evidence index.

It is not verification of how access operates for a particular customer, product, workspace, or incident.

OpenAI’s Enterprise Privacy page shows why product and plan scope matter. It says business data is subject to human review on a service-by-service basis.

For ChatGPT Enterprise, ChatGPT Edu, and ChatGPT for Healthcare, the page states that authorized OpenAI employees may access conversations for purposes such as resolving incidents, recovering conversations with explicit customer permission, or meeting legal requirements.

For ChatGPT Business and stored API data, the same page describes a different boundary. It states that authorized employees may require access for engineering support, investigation of potential platform abuse, or legal compliance. It also identifies specialized third-party contractors, subject to confidentiality and security obligations, who may review content for abuse and misuse.

These statements are more useful than a generic claim that human review is “limited.”

They begin to identify:

• the relevant service;

• the category of person who may access data;

• the purpose of access;

• whether third-party reviewers may be involved.

They also show why evidence from one product or plan should not be applied automatically to another.

The public page does not, by itself, establish every operational detail a buyer may require. It does not fully show the approval workflow, event-level access logs, time limits, reviewed-copy lifecycle, regional access location, customer-visible records, or customer-specific contractual terms.

The source describes an access boundary.

The buyer still needs to determine whether that boundary is sufficient for the intended workflow.

What it does not prove

A human-review statement does not automatically establish:

Data-type coverage. It may not show whether reviewers can access prompts, outputs, uploaded files, audio, transcripts, embeddings, metadata, logs, support tickets, screenshots, or derived records.

Review triggers. It may not distinguish customer-requested support, automated abuse flags, security incidents, quality sampling, legal requests, product evaluation, or internal investigations.

Reviewer identity. It may not show whether access is limited to employees or may also include contractors, subprocessors, support agents, safety teams, engineers, or third-party reviewers.

Access controls. It may not establish whether access requires approval, is time-limited, is logged, is monitored, or is visible to the customer.

Secondary systems. Reviewed content may be copied into support tools, ticketing systems, safety queues, labeling platforms, incident systems, or evaluation datasets.

Retention and deletion. The original content and the reviewed copy may follow different retention periods or deletion processes.

Customer control. It may not show whether the customer can opt out, restrict review, require approval, or receive notice of access.

Contract scope. Public documentation may not establish whether the same commitment applies to the buyer’s product, plan, region, feature, and agreement.

None of these gaps prove that review is uncontrolled.

They show that the available description may not yet establish the complete exposure boundary.

Weak-answer pattern

This is the safety-control blur.

The vendor frames human review as a safety or support measure, and the buyer records that statement as positive assurance.

But the same control can create a separate access path.

A weak review record says:

Human review is limited to safety and support.

That may be directionally useful.

It does not establish what content is accessible, what triggers access, which roles are involved, whether access is logged, where reviewed copies are stored, or whether customers can restrict the process.

Without those details, human review is described.

It is not bounded.

Evidence request

Do not ask only:

“Do you use human review?”

Ask for the review boundary.

A stronger request is:

Please describe every circumstance in which employees, contractors, support personnel, safety reviewers, engineers, subprocessors, or third-party reviewers may access customer content or derived data. For each circumstance, identify the relevant product and plan, data types, access trigger, reviewer role, purpose, approval process, logging, retention, deletion, customer control, and supporting contract term.

Then ask:

• Which data types can be reviewed?

• What events or requests trigger access?

• Which employee, contractor, subprocessor, or third-party roles may access the data?

• Is access approved, time-limited, logged, monitored, and auditable?

• Can the customer restrict review or require approval for support access?

• Are reviewed records copied into support, safety, labeling, evaluation, analytics, or incident systems?

• How long are the original content and reviewed copies retained?

• Does the boundary differ by product, plan, feature, configuration, region, or contract?

That is how human review becomes reviewable.

Review note

A weak review note says:

Human review is limited.

A stronger review note says:

The vendor describes human access for safety, support, abuse investigation, incident response, or operational purposes, but the available evidence does not establish the complete review boundary for the intended workflow. The buyer should confirm which data types may be accessed, what triggers access, which employee or third-party roles are involved, whether access is approved and logged, whether reviewed copies enter secondary systems, how those copies are retained or deleted, which customer controls are available, and which contract terms confirm the boundary.

That note does not say the vendor is unsafe.

It says the exposure path is not yet evidence-complete.

Usage boundary

Until human access is bounded, usage should remain conservative.

Limit use to low-sensitivity internal data. Do not introduce customer-confidential data, regulated data, source code, employee records, privileged business material, or externally relied-upon outputs where uncertainty about human access, support workflows, secondary copies, retention, third-party reviewers, or customer controls would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

Human review can support safety, support, and incident response.

It can also create an access path.

The buyer needs evidence for both.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorAssessment #AIGovernance #ThirdPartyRisk #Privacy #VendorRisk

SOC 2 is useful evidence.

It is not automatically evidence that the AI data path is covered.

A SOC 2 report may support the existence of a tested control environment.

It may still not establish whether the specific AI product, feature, model route, support workflow, retention path, region, and contract boundary match the buyer’s intended use.

That is the evidence gap.

The question is not only:

Does the vendor have SOC 2?

The better question is:

What does the SOC 2 report prove for this AI workflow, and what still needs to be mapped before real data use?

Claim

The vendor says:

“We have a SOC 2 Type II report.”

Or:

“Our platform is SOC 2 compliant.”

Or:

“Our security controls are independently audited.”

These statements may be useful.

They do not, by themselves, establish that the buyer’s actual AI data path is covered.

Why it sounds sufficient

SOC 2 sounds formal.

It usually involves an independent auditor, a defined system, a reporting period, management assertions, control descriptions, testing procedures, and test results.

For a conventional SaaS review, the report may answer important baseline questions about access management, change management, incident response, monitoring, vendor management, and security operations.

That matters.

But an AI vendor review usually asks a narrower question.

The buyer is not simply purchasing “the platform.”

The buyer may be sending prompts, files, meeting audio, transcripts, summaries, source code, customer records, metadata, embeddings, logs, support tickets, or evaluation data into a specific workflow.

That workflow may involve:

• a hosted application;

• one or more model providers;

• retrieval or embedding systems;

• transcription services;

• support and troubleshooting tools;

• safety or abuse-review systems;

• logging and monitoring platforms;

• regional infrastructure;

• customer-selected integrations.

A SOC 2 report may cover some or all of those elements.

It may also apply only to a more narrowly defined system.

The buyer cannot determine that from the SOC 2 label alone.

What it actually proves

A SOC 2 Type II report may establish that specified controls operated within a defined system and reporting period, subject to the scope, criteria, testing, exceptions, and limitations described in the report.

It may support findings such as:

• access-control procedures were documented and tested;

• privileged access was reviewed;

• system changes followed controlled processes;

• relevant systems were monitored;

• incident-response procedures existed;

• selected vendors were subject to review;

• security policies and operational controls were maintained.

These are meaningful evidence points.

They establish something about the audited control environment.

They do not automatically establish the scope of the buyer’s AI workflow.

A public evidence example

The CodeYourCompliance public evidence profile for Harvey records several public assurance surfaces, including Harvey’s Trust Center, security material, DPA references, subprocessor disclosures, and dated SOC 2 and ISO updates.

The profile is an evidence index.

It is not verification that a particular Harvey product, feature, plan, integration, or customer workflow falls within the scope of a specific audit report.

Harvey’s public Trust Center lists SOC 2 Type II among its compliance evidence and states that its current SOC 2 documentation is available through the portal.

Harvey’s security page also says that the company undergoes annual SOC 2 Type II and ISO 27001 audits. The same page describes a wider product environment involving uploaded documents, queries, responses, model providers, regional processing, retention controls, and subprocessors.

Those are useful public evidence points.

They show that independent assurance exists and that the service has multiple data-handling surfaces.

They do not allow a public reviewer to determine the complete audit boundary.

The public Trust Center does not expose the full SOC 2 system description, control tests, exceptions, complementary user-entity controls, reporting period details, or the treatment of every AI feature and external provider without access to the underlying report.

Harvey’s Security Addendum provides an important next step. It states that customers may request the SOC 2 Type II report and, where applicable, relevant penetration-test summaries and data-flow diagrams.

That distinction matters.

The public SOC 2 statement establishes the existence of assurance evidence.

The private report establishes the audited system boundary.

The product documentation and data-flow materials help determine whether that boundary maps to the buyer’s actual AI workflow.

None of these sources should be stretched beyond what it proves.

What it does not prove

A SOC 2 statement does not automatically establish:

Product scope. The report may cover a defined service environment without identifying every AI product, feature, add-in, agent, integration, or beta capability.

Plan and configuration scope. The buyer’s product tier, regional deployment, optional feature, or customer configuration may differ from the environment described in the report.

Model-provider scope. The report may not establish which external model providers process data, what those providers receive, or whether their controls are included or treated as subservice organizations.

Data-type coverage. Prompts, outputs, uploaded files, transcripts, embeddings, metadata, logs, support artifacts, and derived data may follow different paths.

Operational workflows. Support access, abuse monitoring, safety review, troubleshooting, evaluation, and incident response may introduce systems or personnel not obvious from the SOC 2 badge.

Retention and deletion. The report does not automatically establish deletion across active storage, caches, logs, backups, support copies, and downstream providers.

Regional routing. A company-level audit does not prove where each data type is processed or whether routing changes by region, model, fallback path, or integration.

Current applicability. A Type II report covers a historical period. The buyer still needs to consider the evidence date, subsequent changes, bridge coverage, new features, and material changes to the service.

Contract scope. Audit evidence does not replace the DPA, order form, security addendum, regional commitment, or product-specific contractual term.

None of these gaps show that the vendor’s controls are deficient.

They show that the SOC 2 claim is narrower than the decision the buyer is trying to make.

Weak-answer pattern

This is the certification-scope blur.

The buyer asks:

Can we use this AI vendor for this workflow?

The vendor answers:

We have SOC 2 Type II.

That is not a bad answer.

It is an incomplete review record.

The missing question is:

Which parts of this AI workflow fall within the audited system boundary?

Without that mapping, SOC 2 remains valuable source material.

It is not evidence-complete for the intended use case.

Evidence request

Do not ask only:

Can you provide your SOC 2 report?

Ask for scope mapping.

A stronger request is:

Please confirm whether the current SOC 2 report covers the exact product, plan, features, region, and AI workflow under review. Identify how prompts, outputs, uploaded files, transcripts, embeddings, metadata, logs, support artifacts, model-provider routes, integrations, and subprocessors relate to the audited system boundary.

Then ask:

• What legal entity, system, products, and services are included in the report?

• What reporting period does it cover?

• Which AI features and customer plans are included?

• Which model, transcription, embedding, retrieval, or infrastructure providers are involved?

• Are those providers included in the scope, carved out, or addressed through vendor-management controls?

• Are support, troubleshooting, safety, abuse-review, and human-access workflows covered?

• What complementary user-entity controls must the customer operate?

• Have material product or provider changes occurred since the reporting period?

• Which data-flow document, DPA, security addendum, or order-form term confirms the operational boundary?

That is how SOC 2 becomes reviewable.

Review note

A weak review note says:

SOC 2 received. Risk resolved.

A stronger review note says:

The SOC 2 report may support the existence and operation of controls within the vendor’s stated system boundary and reporting period. The available evidence does not yet establish whether the specific AI product, feature, plan, model route, data types, support workflow, retention path, region, and customer configuration are included. Product-specific scope mapping is required before relying on the report for the intended AI workflow.

That note does not reject the vendor.

It identifies where the assurance evidence stops.

Usage boundary

Until the SOC 2 scope is mapped to the intended workflow:

Limit use to low-sensitivity internal data. Do not introduce customer-confidential data, regulated data, source code, privileged business material, or externally relied-upon outputs where uncertainty about product scope, model routing, support access, retention, subprocessors, regional handling, or contractual coverage would materially change the appropriate usage boundary.

That is not approval.

That is review preparation.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

SOC 2 can support that review.

It cannot perform the scope mapping by itself.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

Reply if you want the sanitized sample evidence gap memo.

#AIVendorRisk #AIVendorAssessment #SOC2 #ThirdPartyRisk #AIgovernance