2026-07-29: PASS
2026-07-30: FAIL

The first question is whether the host changed or the test changed. The record cannot answer it.

The package inventory may have moved, but the baseline, freshness rule, policy or evaluator may also have changed. The same evidence can produce a different result when it is judged under a different evaluation context.

With only PASS and FAIL, there is no way to separate a change in the system from a change in the decision conditions. The conclusions were retained. The basis for them was not.

That is why a stored control result is not replayable evidence. An earlier note asked whether audit evidence can survive replay; the harder implementation question is what must be retained for that replay to mean anything.

What must survive the original decision

Replayable evidence is not the same as retained documentation. Replay does not mean running the same scanner again.

A new scan observes the system as it exists now. Replay reconstructs a previous decision from the artifacts that existed when that decision was made.

For that reconstruction to be credible, the retained package must include more than the observation. It must also preserve the schema that admitted the evidence, the context that supplied the threshold or baseline, the policy artifact, the selected decision engine, the implementation version and an integrity digest binding the assessment to the original evidence bytes.

Each artifact answers a different review question.

The evidence shows what the collector observed. The schema shows what input was considered admissible. The context records the rule parameters used at that time. The policy contains the decision logic. Runtime metadata identifies the execution path. The digest shows whether the retained evidence has changed since evaluation.

If any of these elements is missing, part of the decision must be inferred rather than replayed.

A timestamped result may still be useful operationally, but it is only a historical claim unless the decision path can be reconstructed.

The collector should not decide compliance

Consider a read-only collector operating within an explicit audit boundary. It returns the following package inventory:

{
  "packages": [
    {"name": "openssl", "version": "3.0.13"},
    {"name": "python3", "version": "3.12.3"},
    {"name": "curl", "version": "8.5.0"}
  ]
}

This tells us what the collector observed. It does not tell us whether the host complies with a package baseline.

That decision depends on a separate context:

{
  "required_packages": [
    {"name": "openssl", "version": "3.0.13"},
    {"name": "python3", "version": "3.12.3"},
    {"name": "curl", "version": "8.5.0"}
  ],
  "unexpected_package_mode": "ignore"
}

If the required OpenSSL version changes, the same inventory may fail. If unexpected packages are changed from ignore to fail, an additional package may alter the result even though nothing on the host has changed.

The observation and the assessment therefore belong to different layers.

Package names and versions come from collection. Missing packages, mismatched versions and unexpected packages are derived by comparison. Pass or fail is produced only after policy evaluation.

Writing the control result back into the evidence object hides those transitions. The final document becomes easier to read, but harder to inspect. A reviewer can see the conclusion but cannot tell which facts came from the host, which were derived by the replay engine and which depended on policy choices.

Invalid, stale and failed are different states

A control should fail only after admissible evidence reaches the policy and the required condition is not met.

If the evidence does not satisfy its schema, the policy should not evaluate it. A missing timestamp, malformed package entry or incorrect field type is a problem with the evidence path, not proof that the host failed the control.

The assessment should record that distinction:

evidence_status = invalid_evidence
control_status = unknown
decision_executed = false

Evidence can remain intact and still become too stale to support a current conclusion. In this control, stale evidence creates a different problem from either invalid evidence or policy failure. A package inventory may be structurally valid and still be too old for the decision being attempted.

Suppose the context requires evidence collected within the previous 24 hours. An inventory from three days ago cannot support a current package-baseline decision. It does not prove that the host is non-compliant. It proves that the available observation is no longer fresh enough to answer the question.

evidence_status = stale_evidence
control_status = unknown
decision_executed = false

These distinctions are not cosmetic. They imply different actions.

A failed control may require investigation of the host or baseline. Stale evidence requires recollection. Invalid evidence requires examination of the collector, schema or processing pipeline.

A dashboard that reduces all three conditions to red has removed information that operations and audit both need.

Collection and policy execution are separate boundaries

The earlier implementation note moved from evidence principles to an executable control. The public reference artifacts now apply that structure through a fixed replay sequence:

evidence schema gate

-> context schema gate

-> integrity gate

-> freshness gate

-> derived facts

-> policy evaluation

-> assessment output

For the package-baseline example, the comparison stage derives three groups of facts:

• missing_packages

• version_mismatches

• unexpected_packages

The policy then decides how those facts affect the control result.

A missing required package causes failure. A version mismatch causes failure. An unexpected package may be ignored or treated as a violation depending on the explicit context.

The collector does not need to know any of this. Its job is to report the inventory accurately and preserve its collection metadata. Policy belongs later in the pipeline, where its input and execution can be inspected independently.

The same boundary applies to the decision engine.

A built-in evaluator can implement the same decision semantics as a Rego policy, but it does not execute the Rego artifact. That difference must appear in the assessment:

builtin:
decision_executed = true
policy_artifact.executed = false

opa:
decision_executed = true
policy_artifact.executed = true

Without this metadata, a result may appear to have been produced by a policy artifact that was never run. The conclusion could be correct while its provenance is false.

Replay should stop before remediation

The package-baseline control can identify a missing package, an incorrect version or an unexpected installation. It does not install, remove or upgrade software.

That is a deliberate boundary.

Remediation requires broader permissions and produces a different class of evidence: authorization records, execution logs, change results and post-remediation observations. Combining those actions with collection makes it harder to establish what state existed before the tool intervened.

A read-only collector preserves the condition it was asked to observe. A separate remediation workflow may act on the assessment later, but it should produce its own evidence package.

Collection establishes what was seen. Correction changes what will be seen next.

Public reference artifacts

The replay structure described here is implemented in the public codeyourcompliance/evidence-validation-pipeline repository.

The current examples are:

• examples/replayable-tls-control

• examples/os-package-baseline

The TLS example applies the pipeline to certificate expiry. The OS package example applies it to required packages, version mismatches and unexpected software.

Both packages retain the evidence, validation schemas, evaluation context, integrity information, policy artifact and execution metadata needed to inspect how an assessment result was produced. Their tests also show how the pipeline behaves when evidence is stale, malformed or modified after hashing.

These examples do not prove that the underlying systems are compliant. They show that a control decision can be preserved in a form that is more inspectable than a stored PASS or FAIL.

They also stop before remediation. Installation, removal, upgrade and other corrective actions belong to a separate workflow with separate permissions and evidence.

Replayability does not establish that a decision was correct. It establishes whether the basis of that decision can still be examined.

Origin and scope

CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations and technical discussions.

MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.

Related reading

• From Evidence Principles to an Executable Control

• Can Your Audit Evidence Survive Replay?

• Why Audit Evidence Goes Stale

• Documents are not system state.

• Screenshots are supporting artifacts, not proof objects.

• Evidence should be timestamped, source-bound, integrity-checked, and replayable.

• Audit evidence must remain current enough to support a conclusion.

• Automatically generated reports are not accountable audit conclusions.

These ideas have shaped almost every article I’ve written.

But they also raised an uncomfortable question.

If these principles only exist in articles, can they really be tested, challenged, or reproduced?

That question has led to the next stage of the project.

Instead of treating articles as the primary deliverable, I will begin treating each implementation as the primary object. Articles become explanations of that implementation rather than the implementation itself.

The goal is to turn individual evidence principles into small, replayable engineering experiments that anyone can inspect, critique, and improve.

The first reference experiment is intentionally narrow.

TLS endpoint
↓
read-only collection
↓
raw evidence
↓
parsed facts
↓
policy evaluation
↓
assessment result
↓
OSCAL representation

The objective is not to build a complete compliance platform.

The objective is to answer one implementation question:

Can a narrow, replayable implementation preserve the distinction between collection, evidence, policy evaluation, and accountable assessment?

Several boundaries are equally important.

This experiment does not claim that a successful assessment proves overall compliance, nor does it attempt to automate professional judgment.

The purpose is much smaller—and, I believe, much more useful.

If a single evidence principle can be implemented, replayed, inspected, and shown to fail under controlled conditions, it becomes more than an idea. It becomes something other people can verify, criticize, and build upon.

That is the direction CodeYourCompliance is taking next.

Not more articles.

More executable knowledge.

Origin

CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This project does not provide legal, regulatory, audit, certification, or compliance advice.

Project Direction

Future articles will increasingly accompany replayable implementation packages rather than stand alone as methodology notes.

Related Reading

• Compliance Automation Starts at Evidence

• Can Your Audit Evidence Survive Replay?

• What a MAS TRM Checklist Cannot Prove

• A Screenshot Is a Supporting Artifact, Not a Proof Object

• A Generated Report Is Not an Accountable Audit Conclusion

AI vendors often describe retention as a setting:

“Zero data retention is available.”

“Enterprise customers can configure retention.”

“Prompts are deleted after 30 days.”

These statements sound operational. They suggest that the buyer controls the data lifecycle.

But a retention setting is not automatically a retention control.

The real review question is not:

What retention period does the vendor advertise?

It is:

What data lifecycle does the available evidence establish for this product, plan, feature, region, and use case?

Claim

A vendor says:

Enterprise customers can configure prompt retention.

That may prove that a retention feature exists for a defined category of content.

It does not prove the full lifecycle.

Why it sounds sufficient

Retention is often reduced to one number: zero days, thirty days, customer-configurable, or deleted on request.

That makes the answer easy to place in a questionnaire and mark complete.

The problem is that an AI workflow rarely creates only one data object.

One interaction may produce:

• prompts and outputs;

• uploaded files;

• extracted text or embeddings;

• metadata and logs;

• moderation or abuse-review events;

• support artifacts;

• cached copies;

• backup copies.

The vendor may use “retention” to describe visible conversation history.

The buyer may assume the same statement applies to the entire workflow.

That assumption is the evidence gap.

What the statement may prove

If the source is reliable and product-specific, the statement may support a few limited conclusions:

• a retention feature exists;

• administrators may be able to shorten or disable visible history;

• an enterprise plan may offer additional controls;

• the vendor has defined a customer-facing retention option.

Those are useful signals.

They are not evidence that every relevant data object follows the same lifecycle.

The source matters too.

A product setting, FAQ, trust center, support email, contract, and customer-specific commitment do not carry the same weight or scope.

What it does not prove

A configurable retention statement does not automatically prove:

Data-type coverage.
Does the setting cover only prompts and outputs, or also files, embeddings, metadata, logs, support copies, and backups?

Storage-layer coverage.
Deletion from the main application does not prove deletion from caches, indexes, vector stores, support tools, observability systems, or backup environments.

Deletion timing.
“Deleted” may mean immediately removed, queued for deletion, removed from active systems within a stated period, or retained in backups until rotation.

Exception paths.
Data may be retained longer for security investigation, legal obligations, customer support, abuse review, fraud prevention, or service improvement.

An exception may be legitimate.

An undisclosed exception means the advertised period is not the full boundary.

Administrative enforcement.
A setting is weak if changes are not logged, new features bypass it, or different workspaces inherit different defaults.

Product and plan scope.
Retention may differ across API and hosted products, enterprise and consumer plans, regions, integrations, beta features, or model routes.

A company-wide statement does not prove that the intended workflow is covered.

Weak-answer pattern

The vendor describes a retention preference without evidencing the full retention control boundary.

The response provides a duration or setting, but not the covered data types, storage layers, exceptions, deletion timing, administrative roles, audit evidence, or contractual scope.

This does not prove improper retention.

It means the buyer cannot yet determine which lifecycle is being accepted.

Evidence request

A stronger request is not:

What is your retention period?

It is:

Provide a product- and plan-specific retention matrix for prompts, outputs, uploaded files, embeddings, metadata, logs, moderation events, support artifacts, caches, and backups.

For each data type, identify:

• default and configurable retention;

• storage layer;

• deletion timing;

• backup handling;

• exception paths;

• support or human access;

• administrator roles;

• audit-log coverage;

• contractual source.

The buyer should also ask:

• Does a setting change apply to existing data or only new data?

• Can the vendor override the configured period?

• Are changes to retention behaviour communicated to customers?

• Do new AI features inherit the same setting?

• What evidence can the customer retain to show the configured state?

The vendor does not need to expose sensitive architecture.

It does need to establish an operationally usable boundary.

Review note

A review note could state:

The available statement identifies a configurable retention setting but does not establish the complete data lifecycle for the intended workflow. Coverage of files, metadata, logs, embeddings, support artifacts, caches, backups, and exception paths remains unclear. The response is not evidence-complete for data requiring verified deletion or short-lived processing.

Usage boundary

The correct boundary depends on the use case.

A thirty-day period may be acceptable for public marketing material and unacceptable for confidential meeting content.

Zero retention may reduce exposure in one workflow but reduce auditability or incident investigation in another.

The shortest period is not automatically the strongest control.

The control is whether the lifecycle is explicit, scoped, enforceable, observable, and supported by evidence.

Until that boundary is established:

Limit use to low-sensitivity data that does not require verified deletion, strict short-lived processing, or customer-specific retention commitments. Do not expand to confidential, regulated, or customer data until retention and deletion are evidenced by data type, storage layer, exception path, and contract scope.

That is not approval.

That is review preparation.

A retention setting may be useful.

But usefulness is not the same as evidence.

The review unit is not the vendor name.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

Prompt retention can support that review.

It cannot remain a single number inside it.

A retention setting can be part of the evidence.

It is not the whole control.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

I have a sanitized sample evidence gap memo showing how retention, routing, support access, and contract gaps can be recorded in a review file.

Reply if you want the sample.

#AIVendorAssessment #AIGovernance #ThirdPartyRisk #Privacy #VendorRisk

Authentication succeeds.

Configuration data appears in the dashboard.

The control status turns green.

Then the reviewer asks:

Which source produced this value?

Which collector version retrieved it?

When was it observed?

Was the full target scope reached?

Was the data changed during normalization?

Which policy evaluated it?

The integration moved the data.

It did not preserve the answers.

That is the boundary.

What an Integration Proves

A working integration may prove that:

• credentials were accepted

• an endpoint was reachable

• a payload was returned

• field mapping completed

• data reached the destination

That is useful operational evidence.

It does not automatically prove that the transported data is valid audit evidence.

The integration may still omit:

• source context

• expected collection scope

• failed targets

• collector identity and version

• collection timestamp

• transformation history

• integrity status

• evidence freshness

• applicable policy version

Compliance evidence automation is not data movement.

It means collecting evidence from source systems, preserving provenance, validating integrity, evaluating policy, and producing a reviewable evidence package.

Missing Data Must Survive the Integration

Suppose a collector is expected to inspect 100 systems.

It reaches 94.

The integration writes 94 successful records into the compliance platform.

The dashboard reports that all received records passed.

Technically, the integration worked.

The evidence package is incomplete.

The correct state is:

expected targets: 100

observed targets: 94

failed collection: 6

It is not:

evaluated environment: compliant

The six missing systems did not fail the control.

They did not pass it either.

They were not observed.

If that distinction disappears during transport, the evidence has already lost part of its value.

What an Evidence Contract Preserves

An evidence contract defines the minimum structure that must survive collection, transport, normalization, evaluation, and reporting.

evidence_contract:
  source_system:
  collector:
  collector_version:
  collection_method:
  timestamp_utc:
  evidence_type:
  evidence_layer:
  integrity_hash:
  validation_status:
  policy_input_ref:

Each field answers a different audit question.

source_system identifies where the evidence originated.

collector and collector_version identify what process observed the source.

collection_method distinguishes read-only observation from collection that may have changed the assessed system.

timestamp_utc records when the state was observed.

evidence_type defines what kind of object was collected.

evidence_layer separates raw evidence, normalized facts, derived facts, policy results, and audit narratives.

integrity_hash supports detection of post-collection mutation.

validation_status preserves whether the evidence is valid, stale, missing, tampered, unknown, or not machine-verifiable.

policy_input_ref links the evidence to the rule evaluation that consumed it.

Without these fields, the destination may contain data.

It does not yet contain a replayable evidence object.

One Green Status Is Not Enough

A common implementation error is using one status field for the entire pipeline:

{
  "status": "success"
}

What succeeded?

The API request?

The collection?

The schema validation?

The integrity check?

The policy evaluation?

The control?

These outcomes should remain separate.

transport_status: success
collection_status: partial
schema_status: valid
integrity_status: verified
freshness_status: stale
policy_status: not_evaluated
control_status: unknown

A single green status is easy to display.

It is difficult to audit.

Normalization Changes Meaning

Integrations often rename fields, convert timestamps, flatten objects, infer Boolean values, and discard unsupported data.

These transformations may be necessary.

They are not neutral.

Suppose the source returns:

{
  "certificate_not_after": "2026-08-01T00:00:00Z"
}

The integration writes:

{
  "certificate_valid": true
}

The second value is not raw evidence.

It is a derived fact.

The system should preserve the source value, transformation rule, evaluation time, and derived result.

Otherwise the integration replaces evidence with an unexplained conclusion.

OPA Cannot Repair Weak Evidence

OPA can evaluate a defined input against a defined policy.

It cannot determine whether the input was collected from the correct source, whether the target scope was complete, or whether the evidence was modified before evaluation.

A reproducible rule result can still be based on weak evidence.

The policy engine is not the point.

The audit problem starts earlier.

The evidence contract defines what the evaluator is allowed to treat as policy input.

Different Failures Need Different Outcomes

The pipeline should distinguish

An API timeout is not automatically non-compliance.

The defensible chain is:

Control failure should be recorded only when valid evidence was evaluated and the required condition was not met.

Failure states must survive the integration.

Technical companion

An evidence contract boundary note and two synthetic integration examples are available in the CodeYourCompliance evidence-validation-pipeline repository.

Evidence contract boundary:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/docs/evidence_contract.md

Integration without an evidence contract:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/integration_without_evidence_contract.json

Integration with an evidence contract:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/integration_with_evidence_contract.json

These examples do not implement a complete integration governance or evidence integrity model.

They show one boundary:

Successful transport does not establish evidence validity.

Boundary

This material is for evidence structuring, integration design review, and technical review preparation.

It does not provide legal, regulatory, audit, certification, compliance, procurement, or implementation advice.

The goal is not to prove MAS TRM compliance.

The goal is to make the transport-versus-evidence boundary visible before synchronized data becomes accepted audit evidence.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Related reading

Why Audit Evidence Goes Stale

Can Your Audit Evidence Survive Replay?

What a MAS TRM Checklist Cannot Prove

A Screenshot Is a Supporting Artifact, Not a Proof Object

A Generated Report Is Not an Accountable Audit Conclusion

It is not the actual AI data path.

An AI vendor may publish a list of third-party service providers.

That list may show which subprocessors the vendor is permitted to use.

But for a buyer, the review question is narrower:

Which subprocessor touches which data, for which product feature, in which region, under which contract boundary, for which use case, and for how long?

If that is not answered, the subprocessor list remains source material.

It is not evidence-complete for AI vendor review.

Claim

The vendor says:

“We maintain a subprocessor list.”

Or:

“Our current subprocessors are listed here.”

Or:

“We only share customer data with approved subprocessors.”

Or:

“Customers will be notified of subprocessor changes.”

That statement may be useful.

It may support a baseline third-party disclosure process.

But it does not automatically show how data moves through the actual AI workflow.

Why it sounds sufficient

A subprocessor list feels concrete.

It usually includes company names, service descriptions, locations, and sometimes update dates or notification terms.

For a standard SaaS review, this may support a basic third-party processing check.

The buyer can see that cloud infrastructure, analytics, support, security tooling, email delivery, logging, or AI infrastructure providers may be involved.

That matters.

But AI vendor review usually needs more than a list.

The buyer may be sending prompts, files, meeting audio, transcripts, source code, customer records, support tickets, embeddings, logs, metadata, or generated outputs into a specific product workflow.

A static subprocessor list may not show which of those data types each third party touches.

It may not show whether a model provider processes prompts and outputs.

It may not show whether transcription, embedding, retrieval, safety filtering, support access, logging, analytics, or abuse monitoring follow different paths.

The list names possible processors.

It does not explain the operating path.

What it actually proves

A subprocessor list may prove that the vendor discloses third parties involved in service delivery.

It may support claims such as:

The vendor maintains a public or customer-accessible subprocessor list.

The vendor has a process for updating subprocessors.

The vendor identifies certain service providers by name.

The vendor describes general processing functions.

The vendor may provide notice of new subprocessors.

Those are useful evidence points.

But they are not enough by themselves.

A subprocessor list can help identify who may be involved.

It does not automatically prove who is involved in the buyer’s actual AI workflow.

What it does not prove

A subprocessor list does not automatically prove which subprocessors process customer content.

It may not distinguish prompts, outputs, uploaded files, audio, transcripts, summaries, embeddings, metadata, logs, analytics events, support records, abuse events, or security telemetry.

It does not automatically prove which subprocessors are used for a specific product, plan, feature, region, or configuration.

It does not automatically prove whether a model provider receives raw prompts, transformed prompts, retrieved context, files, transcripts, embeddings, outputs, or metadata.

It does not automatically prove whether support tools receive copies of customer content.

It does not automatically prove whether logs, monitoring, analytics, abuse review, or safety systems involve separate processors.

It does not automatically prove retention or deletion across subprocessor systems.

It does not automatically prove whether the customer can object to, restrict, approve, or configure subprocessor use.

This is where the review file becomes weak.

The buyer records:

“Subprocessor list reviewed.”

But the file does not show the actual AI data path.

Weak-answer pattern

This is the list-without-routing pattern.

The vendor provides a subprocessor list.

The buyer treats the list as proof that the data path is understood.

But the list may only say:

Provider A: cloud hosting.

Provider B: analytics.

Provider C: support tooling.

Provider D: AI infrastructure.

Provider E: model services.

That does not answer the workflow question.

Which provider processes prompts?

Which provider stores embeddings?

Which provider receives logs?

Which provider handles transcription?

Which provider performs summarization?

Which provider can access support tickets?

Which provider is used in the buyer’s region?

Which provider applies only to a different feature or plan?

Without that mapping, the subprocessor list is incomplete evidence.

It is a directory.

It is not the data path.

Evidence request

Do not ask only:

“Can you provide your subprocessor list?”

Ask for feature-level routing.

A stronger request is:

“Please map the subprocessors and model providers involved in the specific product, plan, region, and AI workflow under review, including which data types each provider receives, processes, stores, logs, or can access.”

Then ask:

Which subprocessors process customer content?

Which subprocessors process prompts, outputs, uploaded files, transcripts, summaries, source code, embeddings, metadata, logs, or support records?

Which subprocessors support model inference, embedding, retrieval, transcription, summarization, safety filtering, abuse monitoring, logging, analytics, or customer support?

Which providers are infrastructure-only, and which touch customer data?

Does routing differ by product, plan, feature, configuration, or region?

Are model providers included in the subprocessor list?

Are support tools included if customer content can be copied into support workflows?

What retention or deletion commitments apply at each subprocessor?

Can customers object to or restrict specific subprocessors?

Which contract terms confirm the boundary?

That is how a subprocessor list becomes reviewable.

Review note

A weak review note says:

“Subprocessor list available.”

A stronger review note says:

“The vendor provides a subprocessor list, but the list does not by itself establish the actual AI data path for the intended workflow. The buyer should confirm which subprocessors and model providers process each relevant data type, which product features use them, whether routing differs by plan, region, or configuration, whether support and monitoring systems receive customer content, and which contract terms confirm the processing boundary.”

That note does not say the vendor is unsafe.

It says the subprocessor evidence is not yet mapped to the workflow.

Usage boundary

Until subprocessor routing is mapped, usage should stay bounded.

A sample usage boundary:

“Use may remain limited to low-sensitivity internal data while subprocessor and model-provider routing is mapped for the specific product, plan, feature, region, data type, support workflow, logging path, retention layer, deletion process, and contract boundary. Do not use with customer confidential data, regulated data, source code, employee records, privileged business records, or externally relied-upon outputs until the actual AI data path is evidenced.”

That is not approval.

That is review preparation.

A subprocessor list can support the review.

It cannot replace the review.

The review unit is not the vendor name.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A subprocessor list is useful.

But the buyer still needs the actual AI data path.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

I now have a sanitized sample evidence gap memo showing how this looks in a review file.

Reply if you want the sample.

The export is real.

The timestamp is real.

The file hash still verifies.

But the system changed after collection.

The evidence is intact.

The conclusion is not.

That is the freshness problem.

A verified hash does not make stale evidence current.

Evidence can be true and stale at the same time

Compliance teams often treat evidence as a static artifact.

A screenshot was captured.

A report was exported.

A log bundle was saved.

A file hash was recorded.

That may prove something useful.

It does not prove the evidence still supports the conclusion being made today.

Integrity asks whether the evidence changed after collection.

Freshness asks whether the evidence is still recent enough to support the current claim.

Those are different questions.

A stale evidence object can still pass integrity verification.

That is the trap.

Freshness is an evidence property

Compliance evidence freshness is the condition that an evidence object remains recent enough, within its expected validity window, to support the compliance conclusion being drawn from it.

Freshness is not decoration.

It is not a report date.

It is not the date the auditor opened the folder.

It belongs inside the evidence object.

A minimum freshness record should preserve:

evidence_freshness:
  collected_at:
  valid_until:
  freshness_window_seconds:
  freshness_status: fresh | stale | expired | unknown

The collection timestamp tells us when the evidence was observed.

The validity window tells us how long that evidence can reasonably support the claim.

The freshness status tells us whether downstream evaluation should proceed.

Without this, the report can quietly reuse old evidence as if it were still current.

Integrity is not freshness

A hash can prove that an evidence object has not changed.

It cannot prove that the system has not changed.

A TLS certificate export can be hashed.

A firewall rule export can be hashed.

An access review file can be hashed.

A cloud configuration snapshot can be hashed.

If the source system changes after collection, the hash still verifies.

The evidence object is stable.

The world it describes may not be.

This is the boundary:

Both are needed.

One does not replace the other.

Why stale evidence should not become pass or fail

A stale evidence object should not silently produce a clean control result.

It should not be converted into pass.

It should not be converted into fail.

The correct outcome is a freshness classification.

For example:

verified evidence + expired freshness window = stale_evidence

That is not the same as non-compliance.

It means the evidence can no longer safely support the current conclusion.

The control may be working.

The control may have failed.

The evidence is no longer strong enough to decide.

That distinction matters.

Automation should preserve it.

The evidence quality gate

Policy evaluation should not begin with the control.

It should begin with the evidence.

The pipeline should first ask:

Is the evidence intact?

Is the evidence source clear?

Is the collector identified?

Is the collection time preserved?

Is the evidence fresh enough?

Only then should the system evaluate the control expectation.

A simple evidence quality gate looks like this:

evidence object

-> integrity verification

-> freshness classification

-> derived facts

-> policy evaluation

-> audit narrative

If freshness fails, downstream evaluation should stop or explicitly carry the stale evidence status.

The report should not hide the freshness failure.

A stale evidence example

In the CodeYourCompliance evidence validation pipeline, the sample stale evidence object uses a TLS certificate evidence record.

The evidence was collected on July 1.

It was evaluated on July 7.

Its declared freshness window was 24 hours.

The integrity status is still verified.

The freshness status is stale.

The policy result is not pass or fail.

It is:

{
  "input_status": "verified_but_stale_evidence",
  "result": "stale_evidence"
}

That is the correct boundary.

The evidence object survived integrity verification.

It did not survive freshness evaluation.

What the audit narrative should say

A weak audit narrative says:

TLS evidence reviewed. No issue noted.

A better audit narrative says:

That is not legal interpretation.

That is evidence discipline.

The narrative should explain what the evidence can still support.

It should also explain what it can no longer support.

MAS TRM-inspired framing

MAS TRM-inspired does not mean MAS TRM prescribes a freshness field.

It does not mean a specific JSON structure, Rego policy, hash algorithm, or evidence pipeline is required.

The framing is narrower.

If a compliance conclusion depends on technical evidence, the evidence should be recent enough to support that conclusion.

That is an engineering interpretation.

It separates the control claim from the proof material behind it.

Freshness changes the report boundary

Reports often collapse time.

They present collected evidence, reviewer notes, control statements, and conclusions inside one document.

That can make old evidence look current.

A freshness field forces the report to show the time boundary.

It asks:

When was this evidence collected?

How long was it valid for this claim?

When was the conclusion made?

Did the evidence remain fresh at conclusion time?

If the answer is no, the report should say so.

The evidence may still be useful background.

It should not be treated as current proof.

Technical companion

A stale evidence example and a narrow OPA/Rego freshness policy demo are available in the CodeYourCompliance evidence-validation-pipeline repository.

Stale evidence example:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/stale_evidence.json

Evidence freshness policy demo:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/policies/demo/evidence_freshness.rego

These examples do not implement a complete evidence integrity model.

They show one boundary:

verified evidence can still be stale evidence

Boundary

This material is for evidence structuring and technical review preparation.

It does not provide legal, regulatory, audit, certification, compliance, procurement, or implementation advice.

The goal is not to prove MAS TRM compliance.

The goal is to make the evidence freshness boundary visible before stale evidence becomes a clean report conclusion.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Related reading

Can Your Audit Evidence Survive Replay?

What a MAS TRM Checklist Cannot Prove

A Screenshot Is a Supporting Artifact, Not a Proof Object

A Generated Report Is Not an Accountable Audit Conclusion

Tool Approval Is Not Workflow Proof

It can also be a data exposure path.

Both can be true.

An AI vendor may say human review is used for safety, abuse prevention, support, troubleshooting, quality assurance, or model evaluation.

That may sound responsible.

But for a buyer, the review question is narrower:

Who can see customer data, under what condition, for what purpose, for how long, in which system, under which contract boundary, and with what customer control?

If that is not answered, “human review” remains too vague for AI vendor assessment.

Claim

The vendor says:

“Human review may be used for safety.”

Or:

“Authorized personnel may access customer data for support and troubleshooting.”

Or:

“Customer content may be reviewed in limited circumstances.”

That statement may be legitimate.

It may describe an important operational control.

But it does not automatically tell the buyer whether the exposure is bounded.

Why it sounds sufficient

Human review sounds responsible.

It suggests that the vendor is not blindly relying on automation.

It may help detect abuse, investigate suspicious activity, troubleshoot product failures, improve quality, or respond to support tickets.

That matters.

But in AI vendor review, human review is not only a control.

It is also a path through which customer data may become visible to people outside the buyer’s organization.

So the buyer needs more than a general statement.

The buyer needs a boundary.

What it actually proves

A human review statement may prove that the vendor has a process where authorized personnel can inspect certain content or operational records under defined circumstances.

It may support claims such as:

The vendor monitors for abuse.

The vendor investigates safety events.

The vendor can troubleshoot customer issues.

The vendor has support access procedures.

The vendor has internal review workflows.

Those may be useful evidence points.

But they are not enough by themselves.

A statement that human review exists does not show what data is reviewed, when review is triggered, who performs it, where reviewed data goes, how long it is retained, whether access is logged, or whether the customer can restrict it.

What it does not prove

A human review statement does not automatically prove which data types are reviewable.

It may not distinguish prompts, outputs, uploaded files, meeting audio, transcripts, summaries, metadata, logs, embeddings, support tickets, abuse events, evaluation data, or derived records.

It does not automatically prove whether review is triggered by customer request, support ticket, abuse signal, automated flag, sampling, product improvement workflow, safety evaluation, or internal investigation.

It does not automatically prove who can access the data.

It may not distinguish employees, contractors, support agents, safety reviewers, engineers, model evaluators, subprocessors, or third-party reviewers.

It does not automatically prove whether access is approved, logged, monitored, time-limited, or customer-visible.

It does not automatically prove whether reviewed content is copied into support systems, ticketing tools, labeling queues, QA systems, analytics systems, model evaluation datasets, or incident systems.

It does not automatically prove whether customers can opt out.

It does not automatically prove retention, deletion, regional handling, plan differences, or contractual commitment.

This is where the review file becomes weak.

The vendor says “limited human review.”

The buyer records “human review controlled.”

But the file does not show the actual review boundary.

Weak-answer pattern

This is the safety-control blur.

The vendor frames human review as a safety or support control.

The buyer accepts it as positive assurance.

But the same statement may also create an exposure path.

The weak review record says:

“Human review is limited to safety and support.”

That may be directionally useful.

But it is not evidence-complete.

The missing questions are:

What content can be reviewed?

What event triggers review?

Who reviews it?

Can reviewers see raw customer content?

Is access approved and logged?

Can customers disable or restrict review?

Are support copies retained?

Are contractors or subprocessors involved?

Which terms make this enforceable?

Without those answers, human review is not bounded.

It is only described.

Evidence request

Do not ask only:

“Do you use human review?”

Ask for the review boundary.

A better request is:

“Please describe all circumstances where human personnel, contractors, support agents, safety reviewers, engineers, subprocessors, or third-party reviewers may access customer content or derived data, including the data types reviewed, review triggers, access roles, approval process, logging, retention, deletion, customer controls, and contract terms.”

Then ask:

Which data types can be reviewed?

What triggers review?

Which roles can access the data?

Are reviewers employees, contractors, subprocessors, or third parties?

Is access approved, time-limited, logged, and auditable?

Can enterprise customers restrict or disable review?

Is customer approval required for support access?

Are reviewed records copied into support, safety, labeling, QA, analytics, or incident systems?

How long are reviewed records retained?

How are reviewed records deleted?

Does the policy differ by plan, product, region, feature, or contract?

That is how human review becomes reviewable.

Review note

A weak review note says:

“Human review is limited.”

A stronger review note says:

“The vendor describes human review for safety, support, or operational purposes, but the available materials do not establish the full review boundary. The buyer should confirm which customer data types may be reviewed, what triggers review, which roles or third parties may access the data, whether access is logged and approved, whether enterprise customers can restrict review, how reviewed records are retained or deleted, and which contract terms confirm the boundary.”

That note does not say the vendor is unsafe.

It says the exposure path is not yet bounded.

Usage boundary

Until human review is bounded, usage should stay conservative.

A sample usage boundary:

“Use may remain limited to low-sensitivity internal data while human review scope, triggers, access roles, support workflows, logging, retention, deletion, subprocessor involvement, opt-out controls, and contract boundaries are clarified. Do not use with customer confidential data, regulated data, source code, employee records, privileged business records, or externally relied-upon outputs until the human review boundary is evidenced.”

That is not approval.

That is review preparation.

Human review may be useful.

But usefulness is not the same as evidence.

For AI vendor review, the buyer needs to know whether human review is:

a bounded safety process,

a support access path,

a model evaluation path,

a data labeling path,

an abuse monitoring path,

or an undefined exposure path.

Those are different review records.

The review unit is not the vendor name.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

Human review can support that review.

It cannot remain vague inside it.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

I have a sanitized sample evidence gap memo showing how this looks in a review file.

Reply if you want the sample.

SOC 2 is useful evidence.

It is not automatically evidence that the AI data path is covered.

A SOC 2 report may support baseline control posture.

It may still not prove whether the specific AI product, feature, model path, support workflow, retention path, region, and contract boundary match the buyer’s use case.

That is the gap.

The question is not only:

“Does the vendor have SOC 2?”

The better question is:

“What does the SOC 2 report actually prove for this AI workflow, and what still needs to be mapped before real data use?”

Claim

The vendor says:

“We have a SOC 2 Type II report.”

Or:

“Our platform is SOC 2 compliant.”

Or:

“Our security controls are independently audited.”

That statement may be useful.

It is not the same as proving that the buyer’s actual AI data path is covered.

Why it sounds sufficient

SOC 2 sounds formal.

It usually involves an independent auditor, a reporting period, a defined system scope, and tested controls.

For a standard SaaS review, that may answer many baseline security questions.

It can help a buyer understand whether the vendor has controls for access management, change management, incident response, monitoring, vendor management, and security operations.

That matters.

But AI vendor review usually asks a narrower question.

The buyer is not only buying “the platform.”

The buyer may be sending prompts, files, meeting audio, transcripts, summaries, source code, customer records, metadata, logs, embeddings, support tickets, or evaluation data into a specific AI workflow.

A SOC 2 report may cover some of that.

It may cover none of it.

The buyer cannot know unless the report scope is mapped to the actual use case.

What it actually proves

A SOC 2 report may prove that the vendor had a tested control environment for the scoped system during the covered period.

It may support claims such as:

The vendor has access control procedures.

The vendor reviews privileged access.

The vendor manages changes through a controlled process.

The vendor monitors relevant systems.

The vendor has incident response procedures.

The vendor tracks certain third-party service providers.

The vendor has documented security policies.

Those are useful evidence points.

But they are not automatically evidence for the specific AI product, AI feature, model provider route, retrieval layer, data retention path, support access path, or customer use case.

SOC 2 can support the review.

It does not replace the review.

What it does not prove

A SOC 2 report does not automatically prove that the specific AI product is in scope.

It does not automatically prove that the buyer’s product plan or tier is covered.

It does not automatically prove which model provider processes the data.

It does not automatically prove whether prompts, outputs, files, transcripts, summaries, embeddings, logs, or metadata are retained differently.

It does not automatically prove whether customer content enters support queues, abuse review, safety review, model evaluation, analytics, or troubleshooting workflows.

It does not automatically prove whether the subprocessor list maps to the actual AI workflow.

It does not automatically prove whether a “no training” claim applies to all operational data, or only to model training.

It does not automatically prove regional routing.

It does not automatically prove deletion across active storage, logs, caches, backups, support copies, or derived data.

It does not automatically prove the contract boundary.

This is where many review files become weak.

The file contains SOC 2.

But the file does not show how SOC 2 maps to the actual AI data path.

Weak-answer pattern

This is a certification scope blur.

The buyer asks:

“Can we use this AI vendor for this workflow?”

The vendor answers:

“We have SOC 2.”

That is not a bad answer.

It is an incomplete review record.

The missing question is:

“Which part of the AI workflow does this report cover?”

For example:

Does the report cover the AI feature itself?

Does it cover transcription?

Does it cover summarization?

Does it cover retrieval or embedding?

Does it cover the model provider route?

Does it cover support access to customer content?

Does it cover regional processing?

Does it cover the buyer’s plan?

Does it cover the current review date?

If those questions are not answered, SOC 2 remains source material.

It is not yet evidence-complete for the use case.

Evidence request

Do not ask only:

“Can you provide your SOC 2?”

Ask for scope mapping.

A better request is:

“Please confirm whether the SOC 2 report scope covers the specific AI product, plan, and features used in this workflow, including processing of prompts, outputs, files, transcripts, summaries, metadata, logs, embeddings, support artifacts, model provider routes, and subprocessors.”

Then ask:

Which product and plan are covered?

Which AI features are in scope?

Are model providers, transcription services, embedding systems, retrieval systems, or external processors included?

Does the report period cover the current review date?

Are support workflows, troubleshooting access, abuse monitoring, or safety review included?

Are logs and metadata treated as customer data, operational data, security data, or something else?

Which controls apply to retention, deletion, access, and routing?

Which contractual terms confirm the operational boundary?

That is how SOC 2 becomes reviewable.

Review note

A weak review note says:

“SOC 2 received. Risk resolved.”

A stronger review note says:

“The SOC 2 report supports the existence of a tested control environment for the vendor’s stated scope and reporting period. It does not by itself establish whether the specific AI product, feature, model path, data type, region, support workflow, retention layer, or customer use case is covered. Additional product-specific scope mapping is required before relying on the report for this AI workflow.”

That note does not reject the vendor.

It draws the evidence boundary.

It tells the buyer what the report supports and where the report stops.

Usage boundary

Until the SOC 2 scope is mapped to the actual AI workflow, usage should stay bounded.

A sample usage boundary:

“Use may remain limited to low-sensitivity internal data while SOC 2 scope, AI feature coverage, data flow, model routing, subprocessor handling, support access, retention, deletion, and contract boundaries are clarified. Do not expand to customer confidential data, regulated data, source code, or externally relied-upon outputs until those evidence gaps are resolved.”

That is not approval.

That is review preparation.

The review unit is not the vendor name.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

SOC 2 can support that review.

It cannot do the whole job by itself.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

I now have a sanitized sample evidence gap memo showing how this looks in a review file.

Reply if you want the sample.

The model is on the approved list.
Access is limited to approved users.
The policy says sensitive work requires review.
The tool is allowed.

Then it appears inside real work.

It helps write code.
It drafts configuration changes.
It summarizes system tickets.
It prepares incident notes.
It generates compliance narratives.
It suggests access-control updates.

Later, someone asks a simple review question:

What was this AI capability allowed to do at the moment it influenced the workflow?

The organization can show the tool was approved.

It cannot show the workflow proof.

That is the control gap.

Tool approval controls entry

An approved tool list answers one question:

Is this tool allowed to be used?

That question matters.

But it does not answer what happened inside a specific workflow.

It does not prove which task was performed, what data was used, what policy version applied, whether review was required, who accepted the output, or what happened when the AI result failed.

Tool approval controls entry.

Workflow proof controls consequence.

These are different evidence problems.

MAS TRM-inspired framing

This is not legal, regulatory, audit, certification, or implementation advice.

The framing is MAS TRM-inspired in an engineering sense.

The question is narrow:

Can AI-supported work produce evidence that is scoped, timestamped, reviewable, and replayable enough to survive later inspection?

Compliance automation is not about generating nicer reports.

It is about making evidence harder to fake, harder to mutate, and easier to replay.

A tool approval record does not do that by itself.

Permission is not proof

A user may be allowed to use an AI tool.

That does not mean the AI capability should be allowed to perform every task available to that user.

This matters when AI enters environments where work has consequence:

coding tools
ticketing systems
configuration pipelines
access request workflows
incident response notes
audit evidence preparation
data analysis notebooks
local automation scripts

A model that can summarize a document is not the same as a model that can suggest a production change.

A tool that can draft an incident note is not the same as a tool that can classify incident severity.

A coding assistant that can explain a function is not the same as one that can generate a patch.

The capability may look the same from the tool list.

It is not the same from the control perspective.

The missing control fields

Once AI capability enters a workflow, the evidence object needs to move closer to the work.

A useful workflow-use record should preserve at least a few fields:

This is not product architecture.

It is an evidence shape.

The point is not to describe the AI tool in general.

The point is to preserve what the AI capability was allowed to do in a specific workflow, at a specific time, under a specific control boundary.

Without that record, the organization may know the tool was approved.

It does not know whether the workflow action was controlled.

Human review is not evidence by default

Many AI policies rely on human review.

That can be useful.

It is also often too vague.

“Human-in-the-loop” is not proof unless the workflow records what was reviewed, what evidence was visible, who accepted the result, and whether the output moved the workflow forward.

A review checkpoint that leaves no record is not a control.

It is a belief.

Approval route is not decision record.

Policy-as-code needs evidence

OPA or another policy engine can evaluate structured workflow evidence.

It can check whether a task is allowed.

It can check whether a data category is permitted.

It can check whether review is required.

It can check whether a failure condition should stop the workflow.

But it cannot evaluate what the system never recorded.

If task scope is missing, policy evaluation becomes guesswork.

If data category is missing, the boundary cannot be tested.

If policy version is missing, the decision cannot be replayed.

If review status is missing, acceptance cannot be distinguished from silent continuation.

Policy-as-code is not magic.

It executes against evidence.

Failure handling is part of the control

The most revealing question is often not what happens when AI works.

It is what happens when AI fails.

Failure handling is not operational cleanup.

It is part of the control design.

If an AI tool suggests an unsafe configuration, the issue is not only model quality.

The issue is whether the workflow prevents that suggestion from becoming consequence without review.

If an AI-generated audit note sounds correct but has no source reference, the issue is not only hallucination.

The issue is whether unsupported narrative can enter the evidence package.

A serious system does not only log errors.

It prevents error from becoming consequence.

Observation is not remediation

The purpose of workflow proof is not to fix every issue automatically.

It is to make the state of control observable.

If an AI output entered a workflow without review, that is an observation.

If a task used data outside the approved scope, that is an observation.

If no policy version was attached, that is an observation.

If the failure path was missing, that is an observation.

Observation comes first.

Remediation comes later.

Collection is not correction.

The review test

A reviewer should be able to ask:

For this AI-supported workflow action, can we show:

what task was performed
what AI capability was used
what data category was touched
what policy version applied
whether review was required
whether review occurred
who owned the decision
what output moved forward
what failure condition was checked
what evidence hash protects the record

If the answer is no, the organization may have tool approval.

It does not yet have workflow proof.

Closing boundary

Approved use is a permission record.

Workflow proof is an evidence record.

The approved tool list controls entry.

It does not prove task scope, data boundary, review condition, policy version, failure handling, or decision ownership inside the workflow.

That proof has to be captured closer to the work.

Tool approval is not workflow proof.

That is the boundary.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This article is not legal, regulatory, audit, certification, compliance, or implementation advice.

Technical Companion

A minimal workflow proof boundary note and synthetic evidence examples are available in the CodeYourCompliance evidence-validation-pipeline repository.

Boundary note:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/docs/workflow-proof-boundary.md

Synthetic examples:

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/minimal_workflow_proof_object.json

https://github.com/codeyourcompliance/evidence-validation-pipeline/blob/main/examples/invalid_workflow_proof_missing_review_owner.json

These examples do not implement a workflow-proof engine. They show the minimum evidence shape for separating tool approval from workflow proof.

Related Reading

• Go-Live Is Not Workflow Evidence

• Can Your Audit Evidence Survive Replay?

• What a MAS TRM Checklist Cannot Prove

• A Generated Report Is Not an Accountable Audit Conclusion

I have started a public side project under CodeYourCompliance:

https://signal.codeyourcompliance.com

The Trust Signal Directory tracks public evidence surfaces that appear across SaaS, AI, and B2B software vendors.

Examples include:

• Trust Centers

• Security pages

• Privacy policies

• Data Processing Addendums

• Subprocessor pages

• SOC 2, ISO 27001, HIPAA, GDPR, and related statements

• AI data usage, responsible AI, and model safety pages

• Enterprise security features such as SSO, SCIM, RBAC, audit logs, and data residency

• Security, privacy, compliance, GRC, and legal operations hiring signals

The purpose is not to rank vendors.

The purpose is to observe public evidence surfaces and ask a more specific question:

What might this public change signal about trust maturity, compliance pressure, enterprise readiness, procurement readiness, or vendor review expectations?

A single page change is not proof of buying intent. A Trust Center does not automatically mean a company is looking for a new compliance tool. A privacy policy update may be routine legal maintenance. A SOC 2 page may indicate the company is already past the readiness stage.

That is why the directory is evidence-first and cautious by design.

Each useful signal should eventually answer:

• What changed?

• Where is the public evidence?

• Why might it matter?

• What could make it a false positive?

• What should be watched next?

The public site will start small.

The first version will focus on methodology, signal taxonomy, sample company profiles, and public examples. More company profiles will be added only after manual review.

The private research layer remains separate. Buyer-specific fit ratings, custom watchlists, recommended GTM actions, and outcome feedback are not published in the public directory.

For now, the goal is simple:

Build a clean public evidence layer for tracking trust and compliance signals, then test whether these signals can become useful GTM intelligence for people selling into compliance, security, privacy, AI governance, vendor risk, and enterprise readiness workflows.

You can view the initial public alpha here:

https://signal.codeyourcompliance.com

It is not the review conclusion.

A vendor may have a security page, privacy documentation, subprocessors page, compliance reports, AI data-use statement, DPA, retention FAQ, and product security overview.

That does not mean the buyer has completed an AI vendor risk assessment.

The review question is narrower:

Which claim does this source support, for which product, plan, use case, data type, region, contract terms, and evidence date?

That is the gap.

Claim

The vendor says:

“See our trust center.”

Or:

“Our security and compliance documentation is available in our trust center.”

Or:

“You can find our SOC 2, DPA, subprocessors, privacy policy, and security information there.”

That response may be useful.

It is not enough by itself.

A trust center can contain evidence sources.

It does not automatically convert those sources into a reviewable evidence file for the buyer’s actual AI workflow.

Why it sounds sufficient

Trust centers feel organized.

They usually collect the documents a reviewer expects to see:

For a busy buyer, this can look like the answer.

The vendor has a portal.

The documents are in one place.

The compliance logos are visible.

The buyer can download files.

That can create the impression that the vendor risk review is mostly complete.

But a trust center is still vendor-controlled source material.

It does not answer the buyer’s operating-context question by itself.

What it actually proves

A trust center may prove that the vendor has published relevant documentation.

It may show that the vendor has some formal security, privacy, compliance, and procurement materials available.

It may help identify where claims are written.

It may provide useful evidence sources for a review file.

It may help the buyer locate:

where the vendor makes its data-use promises,

which certifications exist,

which subprocessors are listed,

which DPA terms are available,

which security controls are described,

which privacy commitments are public,

and which documents may need to be requested privately.

That is useful.

But it is only the starting point.

The trust center helps locate evidence sources.

It does not decide whether those sources support the actual claim, product, plan, data path, or use case under review.

What it does not prove

A trust center does not automatically prove that a specific AI product is covered.

This is where reviews become weak.

The file says:

“Trust center reviewed.”

But the file does not show which claim each source supports.

Weak-answer pattern

This is the trust center showroom pattern.

The vendor points the buyer to a polished evidence showroom.

The buyer receives many documents.

But the review still lacks claim-to-source mapping.

The weak review record looks like this:

“Vendor has trust center. SOC 2 available. DPA available. Subprocessor page available. Security documentation available.”

That may be true.

But it does not answer:

Which exact claim is supported?

Where is the source?

Is the source current?

Is the source contractual, audited, official documentation, policy text, or marketing language?

Does it cover the product?

Does it cover the plan?

Does it cover the AI feature?

Does it cover the data type?

Does it cover the use case?

Does it cover the region?

Does it cover the support workflow?

Does it cover model routing?

Does it cover retention and deletion?

Does it cover subprocessors?

Does it cover human review?

Without that mapping, the trust center is not evidence-complete.

It is a source library.

Evidence request

Do not ask only:

“Can you share your trust center?”

Ask for claim-specific mapping.

A stronger request is:

“Please identify which trust center materials support each AI data-use, security, privacy, retention, subprocessor, support access, human review, and model-training claim for the specific product, plan, region, and use case under review.”

Then ask:

Which document supports the claim that customer prompts and outputs are not used for model training?

Where is the retention period for prompts, outputs, uploaded files, logs, metadata, embeddings, and support records documented?

Which subprocessors process customer content for this product and feature?

Which subprocessors process only metadata, infrastructure telemetry, support data, or security logs?

Is human review possible for prompts, outputs, files, abuse events, support tickets, or troubleshooting records?

Can the customer configure retention, deletion, data residency, and support access?

Which claims are contractual?

Which claims are in public policy pages only?

Which claims are covered by audited reports?

Which claims require private trust center material, SOC 2 detail, security questionnaire response, order form language, or customer-specific terms?

The goal is not to collect more documents.

The goal is to map claims to evidence.

Review note

A weak review note says:

“Trust center available.”

A stronger review note says:

“The trust center provides source material, but it does not by itself establish that the vendor’s claims are evidence-complete for the intended AI workflow. Each relevant claim should be mapped to a source, evidence type, product scope, plan, data type, region, contract boundary, and evidence date before the buyer relies on the claim for customer, confidential, regulated, or externally relied-upon data use.”

That note does not reject the vendor.

It draws the evidence boundary.

It tells the buyer what the trust center supports and what still needs to be checked.

Usage boundary

Until the trust center materials are mapped to the actual use case, usage should stay bounded.

A sample usage boundary:

“Use may remain limited to low-sensitivity internal data while trust center materials are mapped to the specific AI product, plan, feature, data type, model path, subprocessor chain, support access path, retention period, deletion control, regional route, and contract boundary. Do not expand to customer confidential data, regulated data, source code, employee records, or externally relied-upon outputs until the relevant claims are tied to reviewable evidence.”

This is not approval.

It is review preparation.

The review unit is not the vendor name.

The review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date.

A trust center can support that review.

It cannot do the whole review by itself.

Boundary

This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

The goal is not to approve or reject a vendor.

The goal is to make the evidence gap visible before real data use.

I now have a sanitized sample evidence gap memo showing how this looks in a review file.

Reply if you want the sample.

The platform is live.
The automation is deployed.
The audit findings are closed.
Training completion is above target.
The policy has been approved.
The programme reports 85% complete.

Then someone from operations says the quiet part.

“My team worked all weekend fixing what the automation broke.”

That is where the audit problem starts.

Not because the dashboard is fake.

Because the dashboard is measuring delivery progress.

Operations is measuring whether the workflow actually works.

Those are not the same thing.

A system can be delivered and still fail to change the operating reality. A control gap can be remediated and still leave the process weak. An AI pilot can show better detection and still fail at scale because data ownership, evidence capture, escalation, and review paths were not ready.

Go-live is delivery.

It is not workflow evidence.

Delivery evidence is not workflow evidence

Most transformation reporting is built around delivery objects.

The platform went live.
The dashboard was built.
The policy was approved.
The training was completed.
The audit finding was closed.
The AI model was deployed.
The control remediation was marked complete.

These records have value.

They prove that the programme produced something. They help track spend, schedule, implementation, and governance commitments. They support programme control.

But they do not prove that the workflow changed.

A closed finding does not prove operating capability improved.

A live dashboard does not prove decisions became better.

A deployed automation does not prove manual effort reduced.

A trained user population does not prove the new process became the normal path.

A successful pilot does not prove the organisation can operate the control at real volume.

The delivery record answers one question:

The workflow evidence answers a different question:

That is the boundary.

The workaround is evidence

When a transformation fails to land, the first sign is often not a red dashboard.

It is a workaround.

A spreadsheet continues to run beside the platform.
An analyst keeps a manual tracker because the new workflow misses exceptions.
A team rechecks AI output because no one trusts the source data.
A manager approves in the system but makes the real decision in a side channel.
Operations absorbs the exception load while the programme reports adoption.
Evidence is reconstructed later because capture was not designed into the workflow.

The workaround is not just noise.

It is evidence that the formal process has not absorbed the work.

That does not mean every workaround is bad. Some workarounds are rational responses to an incomplete control path. They show where the operating model could not yet carry the real workflow.

This is why transformation reporting often stays green while the organisation feels worse.

Every layer translates reality upward.

“Control failures” becomes “remediation underway.”
“Operational pressure” becomes “resource challenge.”
“Manual evidence reconstruction” becomes “temporary support activity.”
“Benefits delayed” becomes “value realisation in progress.”

None of those phrases are necessarily false.

They are just not workflow evidence.

They protect the programme narrative. They do not prove the change landed.

What workflow evidence should show

Workflow evidence is narrower than a transformation story.

It should show what changed, where it changed, how it was measured, and who owns the result.

A useful workflow evidence object should answer:

This is not about making reporting prettier.

It is about separating delivery status from operating proof.

Data alone is not evidence.

A dashboard alone is not accountability.

Delivery alone is not transformation.

MAS TRM-inspired, not MAS TRM-prescribed

This is where MAS TRM-inspired engineering is useful.

Not as legal advice.
Not as regulatory interpretation.
Not as certification language.

The useful lesson is narrower.

Technology risk governance becomes stronger when evidence is collected in a way that is hard to fake, hard to mutate, and easy to replay.

That means read-only collection where possible.
Timestamped evidence.
Collector metadata.
Policy or control version.
Integrity reference.
Control result.
Exception state.
Owner.
Audit narrative.

The implementation is not prescribed by MAS TRM.

The engineering interpretation is that control proof should survive later review.

A report may persuade in a steering committee.

Evidence has to survive challenge.

The hard question

Before calling a programme transformed, ask one question:

What proves the workflow is better?

Not what was delivered.
Not what was closed.
Not what was approved.
Not what was trained.
Not what went live.

What proves the workflow changed?

Did the workaround disappear?
Did manual evidence effort reduce?
Did decision time improve?
Did exception handling improve?
Did false positives fall?
Did escalation become clearer?
Did the control result become reproducible?
Did the owner become clear?
Can the decision path be replayed?

If the answer is not available, the programme may still be valuable.

But it is not yet proven transformation.

It is delivered change without workflow proof.

Go-live is delivery.
Closed finding is remediation.
Workflow proof is transformation.

That is the boundary.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This article is not legal, regulatory, audit, certification, compliance, or implementation advice.

Related Reading

• A Screenshot Is a Supporting Artifact, Not a Proof Object

• What a MAS TRM Checklist Cannot Prove

• Can Your Audit Evidence Survive Replay?

• Compliance Automation Starts at Evidence

The tool maps controls.

It summarizes gaps.

It drafts findings.

It produces a clean report.

Then the reviewer asks one question:

Who concluded that the control was operating?

The report cannot answer that by itself.

That is the failure mode.

A generated report is not an accountable audit conclusion.

It may be useful.

It may reduce drafting work.

It may make the review file easier to read.

But it does not automatically prove that evidence was valid, interpreted correctly, reviewed under the right standard, and accepted by a responsible reviewer.

That is the boundary.

Evidence processing is not evidence judgment

AI can support audit work.

It can collect records.

It can map controls.

It can compare evidence.

It can summarize exceptions.

It can draft findings.

None of that is the problem.

The problem starts when evidence processing is mistaken for evidence judgment.

Evidence processing asks:

Evidence judgment asks:

Those are different acts.

The tool can assist the first.

It should not silently inherit the second.

MAS TRM-inspired does not mean audit automation

CodeYourCompliance uses MAS TRM-inspired engineering language.

That does not mean MAS TRM prescribes this implementation.

It does not mean the output is legal, regulatory, audit, or certification advice.

The point is narrower.

MAS TRM-style control thinking gives a useful engineering structure:

That structure prevents one common failure.

A report says the control passed.

But the evidence object cannot show what was collected, when it was collected, from where, by what collector, under which schema, and against which acceptance standard.

That is not a writing problem.

That is an evidence architecture problem.

A report is narrative

A report is narrative.

Evidence is proof material.

A control is not proven because a paragraph says it was satisfied.

A control is supported when the underlying evidence can survive review.

A minimal evidence object should preserve:

evidence_id: ev-2026-001
control_id: access-review-001
source_system: identity_platform
collector_type: read_only_export
collected_at: 2026-06-13T09:20:00Z
artifact_type: access_export
artifact_hash: sha256:...
schema_version: evidence.v1
policy_version: access_review_policy.v1
evaluation_result: pass | fail | manual_review | invalid_evidence
review_status: pending | accepted | rejected

This does not make the conclusion automatic.

It makes the conclusion reviewable.

The report can explain the finding.

The evidence package should show:

Reports persuade.

Evidence survives.

The reviewer still owns the conclusion

AI-assisted audit work can make weak review files look strong.

The report may sound complete.

The finding may sound precise.

The control mapping may look convincing.

But if no accountable reviewer has accepted the evidence, the conclusion is not anchored.

The review file should separate five layers:

Do not collapse them.

A collector collects.

A policy evaluates.

A model summarizes.

A reviewer concludes.

A report narrates.

Each layer has a different job.

If the model invents certainty, the narrative becomes dangerous.

If the reviewer does not accept the conclusion, the report is only a draft.

Invalid evidence is not a failed control

A generated report often hides another problem.

It treats every bad artifact as a bad control.

That is wrong.

Invalid evidence is not the same as control failure.

Example:

Control: privileged access must be reviewed quarterly.

Artifact: screenshot of admin users.

Problem: screenshot has no export timestamp, no source binding, no hash, and no review-period marker.

The correct result is not automatically:

The correct result may be:

A control failure says the system state did not meet the control requirement.

Invalid evidence says the artifact cannot prove the system state.

Those are different findings.

If evidence is invalid, the report should not pretend to know the control outcome.

It should say:

That is not softer.

It is more precise.

The report comes late

A useful compliance automation pipeline should not start by drafting the final report.

It should start by preserving the evidence path.

The report comes late.

The evidence comes first.

If the report is generated before evidence quality is known, the system produces confidence before proof.

If the evidence object is weak, the narrative should remain weak.

If the policy evaluation returns manual_review, the report should not say pass.

If the artifact hash changes, the evidence package should flag mutation.

If collector metadata is missing, the evidence should be downgraded.

The tool is not the point.

The audit structure is the point.

A better generated report

A weak generated report says:

A better report says:

This is less polished.

It is more honest.

It separates machine evaluation from human conclusion.

It separates evidence from narrative.

It separates control support from final acceptance.

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com
GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This is not legal, regulatory, audit, certification, compliance, or implementation advice.

Related Reading

• A Screenshot Is a Supporting Artifact, Not a Proof Object

• What a MAS TRM Checklist Cannot Prove

• Can Your Audit Evidence Survive Replay?

• Compliance Automation Starts at Evidence

Vendor claim is not evidence.

“We do not train on your data” is one of the most common AI vendor statements. It is also one of the easiest to over-credit.

The claim may be true. It may be helpful. It may even appear in a contract.

But by itself, it is not evidence-complete.

Training is only one possible use of customer data. The real review question is:

That means looking past the slogan and asking about retention, logging, review workflows, support access, metadata, subprocessors, model providers, deletion, audit logs, tenant settings, and contract scope.

Claim

Or:

Or:

These statements sound clear. They are not all the same.

One may cover customer content. Another may cover prompts and outputs only. Another may cover training, but say nothing about logs, telemetry, evaluation, support access, or product improvement.

The exact wording matters.

So does the source.

A public FAQ is not the same as a customer contract. A product page is not the same as a data processing addendum (DPA). A trust center statement is not the same as tenant-level evidence.

Why it sounds sufficient

It answers the question most buyers have been trained to ask:

That is a real concern. If proprietary or sensitive data can be used for model training, the buyer has a serious problem.

So when the vendor says no, it feels like the main issue is closed.

That is where reviews go off track.

The claim addresses one use. It does not explain the operational data path. A vendor may not train on customer data and still retain prompts, keep logs, allow support review, or send data through model providers and subprocessors.

What the claim can support

A no-training statement can support a narrow point:

That is useful. It is just not enough.

The strength depends on where it appears. A contract or product-specific term is stronger than a marketing page. A signed response is stronger than a generic FAQ.

It is also only as good as its scope. If the statement does not define the covered data, product, plan, exceptions, and document source, it remains incomplete.

At minimum, the buyer should be able to answer:

Without that, the claim is reassuring but thin.

What it does not prove

A no-training claim does not automatically prove:

• prompt retention period

• logging and telemetry scope

• review and support access boundaries

• metadata handling

• support access boundary

• subprocessor data path

• model provider data path

• deletion controls

• audit log availability

• tenant setting defaults

• customer-specific contract coverage

That is the gap. The vendor answered training use. The buyer still does not know the full data path.

Weak-answer pattern

The weak-answer pattern is a narrow promise presented as if it closed the review.

The vendor says:

But the answer does not address the rest of the path:

The statement may be true. It is still too narrow. A strong answer maps the claim to data categories, retention, third parties, customer controls, and contract scope.

Evidence request

Do not ask the vendor to restate the slogan. Ask for evidence that maps the claim to actual handling:

That is a much better question than “do you train on our data?”

Review note

Usable review language:

That note records the evidence gap without pretending the claim proves more than it does.

Usage boundary

Until the missing evidence is resolved, keep the usage boundary narrow:

That is not approval or rejection. It is a review boundary.

Bottom line

“We do not train on your data” may be true. It may be useful.

It is still not the review.

The buyer still needs to know what enters the product, where it goes, how long it stays, who can access it, which third parties touch it, what can be deleted, what can be audited, and which commitments actually apply to the buyer’s product, plan, tenant, and contract.

Vendor claim is not evidence.

The work is turning a narrow public claim into a mapped evidence request, a usable review note, and a conservative usage boundary.

This is part of the AI Vendor Evidence Gap Pack series: vendor claim → evidence source → evidence gap → buyer question → usage boundary.

Boundary

This article is for evidence structuring and review preparation.

It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

Examples are illustrative unless separately validated for a specific organization and use case.

The problem is not that teams lack questions. The problem is that vendor answers are not converted into evidence requests, weak-answer patterns, red flags, and residual risk language.

Opening direction

Most AI vendor reviews start with a questionnaire.

That is reasonable. Questionnaires create structure. They force vendors to answer in writing. They give procurement, security, legal, risk, and audit teams a shared object to review.

But the questionnaire is not the control.

A vendor can answer every question and still leave the buyer with little usable evidence.

The gap appears after the answer arrives.

The vendor says:

We do not train on your data.

The review question should not stop there.

It should ask:

What does training mean?

Does that include fine-tuning?

Does that include evaluation?

Does that include abuse monitoring?

Does that include support review?

What is retained?

What is logged?

Which subprocessors touch the data?

Can the customer export evidence later?

The claim may be true. But it is not yet evidence-complete.

Core argument

A questionnaire collects vendor statements.

An evidence review converts those statements into reviewable objects.

Those objects include:

• evidence requests

• weak-answer patterns

• red flags

• review notes

• residual risk language

That conversion step is where many AI vendor reviews are weakest.

Why AI vendors make this harder

Traditional vendor risk templates were built for more stable control surfaces.

They work reasonably well for questions about:

• access control

• encryption

• incident notification

• business continuity

• SOC 2 availability

• subprocessors

• data processing terms

AI vendors add new ambiguity.

The ambiguity is not always in the security layer. It is often in the behavior layer.

Examples:

• What data enters the model context?

• What data is retained in logs?

• What data is used for evaluation?

• What changes when the model version changes?

• Can the customer identify which model produced a prior output?

• What actions can an AI agent take?

• What approvals exist before an external action is triggered?

• Can logs be exported for audit or incident reconstruction?

A generic questionnaire can ask about these issues, but the harder work is deciding whether the answer is evidence-complete.

Example: polished answer, weak evidence

Vendor answer:

We use industry-leading safeguards to ensure customer data is protected and is not used to train our models.

This answer sounds reassuring.

But as evidence it is weak unless it is supported by:

• contractual language

• retention periods

• logging scope

• support access boundaries

• subprocessor handling

• opt-out configuration

• evaluation and fine-tuning scope

• exportable records

The issue is not whether the vendor is bad.

The issue is whether the buyer can rely on the claim later.

The review shift

The review should move from:

Did the vendor answer the question?

To:

Can the vendor answer be preserved as evidence?

And then:

If the evidence is weak, what residual risk language should appear in the review file?

What AI Vendor Risk Pack is trying to do

AI Vendor Risk Pack is a practical attempt to make that conversion easier.

It starts with common vendor claims and asks:

What evidence should be requested?

What answer would be weak?

What should be treated as a red flag?

What review note should be preserved?

What residual risk language might be needed?

It is not a certification system.

It is not a vendor rating.

It is not legal or audit advice.

It is a review aid for turning AI vendor claims into evidence-aware risk language.

Possible closing

The next phase of AI vendor risk will not be won by longer questionnaires.

It will be won by better evidence conversion.

The real question is not whether the vendor gave an answer.

The question is whether the buyer can later explain why that answer was accepted.

CTA draft

I am building a lightweight AI Vendor Risk Pack under CodeYourCompliance.

The first version focuses on five domains:

1. Data use.

2. Model change and governance.

3. Security and logging.

4. Human approval and agent permission.

5. Auditability and evidence export.

The goal is simple:

Turn AI vendor claims into evidence requests, weak-answer patterns, red flags, review notes, and residual risk language.

The next article applies this to the most common AI vendor claim: “We do not train on your data.”

Read next: Vendor Says It Does Not Train on Your Data. What Evidence Should You Ask For?

One shows TLS enabled. One shows logging configured. One shows a user table. One shows a vendor portal confirmation. One shows a green dashboard.

The folder looks complete.

Then the reviewer asks:

What exactly does each screenshot prove?

Was it captured from the authoritative system? Which account and environment were shown? Was it taken before or after remediation? Was the full scope visible? Was it used for policy evaluation?

The images still look useful.

The proof boundary is unclear.

A screenshot can show what appeared on a screen. It does not automatically prove the underlying system state, complete scope, or continued operation of a control.

MAS TRM is the context. It does not prescribe this evidence classification or pipeline.

MAS TRM-inspired means engineering interpretation.

A screenshot proves a view

A screenshot may support a narrow claim:

At capture time, this interface displayed this information.

That can be useful.

The problem starts when the claim becomes larger than the image.

A screenshot usually cannot establish by itself:

• that the source was authoritative;

• that all in-scope systems were included;

• that the page was current;

• that the expected account or environment was used;

• that the image was not cropped or edited;

• or that it was used for policy evaluation.

A view is not the full system state.

Screenshot is a role, not a tier

Treating every screenshot as secondary and every machine export as primary is too simple.

A machine-generated file can still be stale, incomplete, mis-scoped, or collected from the wrong source.

A screenshot can sometimes be the best available evidence when a system exposes no export or API.

The correct question is:

What claim must this artifact support, and does it carry enough context for that claim?

primary_evidence
Evidence directly used to evaluate a defined condition.

supporting_artifact
Context that helps explain evidence or narrative.

manual_claim
An assertion without independent source-bound observation.

insufficient_evidence
An artifact that does not cover the required claim or scope.

invalid_evidence
An artifact whose integrity, identity, provenance, or linkage cannot be relied upon.

File type does not decide the category.

Evidence requirements do.

Context makes the role inspectable

A screenshot used as evidence should be packaged with context:

artifact_type: screenshot
target_id: production-load-balancer
source_system: load-balancer-admin-portal
environment: production
captured_at: 2026-06-01T10:31:00Z
captured_by: reviewer-01
capture_method: controlled_manual_capture
scope: listener-443
artifact_hash: sha256:...
related_evidence_ref: evidence/tls-runtime-observation.json
policy_result_ref: policy-results/tls-cert-valid.json

These fields do not prove that the screenshot is true.

They make its role inspectable.

The hash only helps show that the preserved image matches the image that was sealed. It does not prove that the screen was genuine, current, complete, or authoritative.

For the broader model, see Compliance Automation Starts at Evidence.

Screenshots often lose scope

A screenshot captures one page, one account, one filter, and one point in time.

A screenshot of five administrators does not prove there are only five administrators.

A green dashboard tile does not prove every underlying check succeeded.

A certificate page does not prove every production endpoint presents the same certificate.

What population was expected? What population was observed? What was excluded?

A screenshot can be accurate and still insufficient.

Capture is not correction

A control owner notices that logging is disabled. Logging is enabled. A screenshot is taken.

The screenshot may support post-remediation verification.

It cannot replace the original observation, remediation record, approval, and later verification.

For that boundary, see Read-Only Collection as an Audit Boundary.

The report should not upgrade the artifact

A reporting system can place a screenshot under the correct control, add a caption, and mark the row complete.

None of that enlarges the evidentiary scope of the screenshot.

The report should state the role:

The screenshot shows the interface state visible during review. The policy result was derived from the referenced source-bound evidence object.

Where no stronger source exists:

The screenshot is the available evidence for the displayed state. Its scope is limited to the identified page, account, environment, and capture time.

A checklist should point to evidence requirements, not merely request uploads. See What a MAS TRM Checklist Cannot Prove.

The replay test

A later reviewer should be able to identify the supported claim, source, capture time, capture method, scope, file integrity, and linked policy result.

That does not recreate the live system.

It recreates the evidence path.

For that test, see Can Your Audit Evidence Survive Replay?.

The position

A screenshot is not useless.

It is not automatically weak.

It is also not automatically a proof object.

It may prove what a screen displayed. It may support human review. It may be the only available observation.

But its claim must remain bounded by provenance, scope, capture method, integrity record, and linkage to evaluated evidence.

Screenshot is an artifact.

Evidence is a supported claim.

Report is narrative.

Do not let the report enlarge the claim.

Related Reading

• What a MAS TRM Checklist Cannot Prove

• Compliance Automation Starts at Evidence

• Read-Only Collection as an Audit Boundary

• Can Your Audit Evidence Survive Replay?

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and public discussion.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This material is not legal, regulatory, audit, certification, implementation, or compliance advice.

Every row is marked complete.

The control owner is assigned.

Evidence links are attached.

A report is prepared.

Then a reviewer asks:

When was the TLS certificate state observed?

Nobody answers cleanly.

Which endpoint was inspected?

Who collected the evidence?

Was the required scope complete?

Did the collector change the target?

Has the evidence changed since collection?

Did the policy result evaluate the same evidence object?

The checklist still looks complete.

The evidence chain does not.

That is the failure mode.

A checklist can organize compliance work. It cannot by itself establish that a technical condition was true at a specific time.

MAS TRM is the context. It does not prescribe this checklist, evidence schema, collector, or policy pipeline.

MAS TRM-inspired means engineering interpretation.

A checklist is a coordination object

A checklist is useful.

It can record:

• the control topic;

• the assigned owner;

• the review status;

• the requested artifact;

• and the completion date.

A signed checklist may also show that an attestation, review, or approval occurred.

That is evidence of a process action.

It is not the same as evidence of the underlying system state.

A row marked “TLS certificate valid” does not establish:

• which certificate was observed;

• which endpoint presented it;

• when it was observed;

• whether all in-scope endpoints were checked;

• or whether the evidence came from an authoritative source.

The checklist records the workflow around the claim.

The evidence object supports the claim itself.

The green status is a narrative state.

It is not automatically a technical state.

The control may operate on some systems but not the full scope.

It may have operated at the time of review but failed later.

It may have been remediated before the original failure was preserved.

A checklist cannot resolve those questions without linked evidence requirements.

Each row needs an evidence contract

The useful extension to a checklist is not another comments column.

It is an evidence contract.

For “TLS certificate satisfies the approved expiry policy,” the evidence requirement might include:

target_id: production-load-balancer
evidence_type: tls_certificate_observation
observed_at: 2026-05-25T10:31:00Z
collector_id: tls-cert-collector
collector_version: 0.1.0
collection_status: success
certificate_not_after: 2026-09-01T00:00:00Z
integrity_hash: sha256:...
policy_result_ref: policy-results/tls-cert-valid-2026-05-25.json

Each field answers a different challenge.

target_id identifies what was inspected.

observed_at anchors the observation in time.

collector_id identifies the process that produced the output.

collection_status prevents failed collection from silently becoming a control result.

integrity_hash helps detect mutation after sealing.

policy_result_ref links the conclusion to the evidence actually evaluated.

The schema does not prove control effectiveness.

It makes the basis of the judgment inspectable.

For the upstream model, see Compliance Automation Starts at Evidence.

An attachment is not automatically evidence

A checklist may accept anything that can be uploaded.

A screenshot.

A PDF export.

An email confirmation.

A spreadsheet.

A meeting note.

These artifacts may support a review.

They do not automatically establish provenance, scope, freshness, integrity, or source authority.

A screenshot may show a screen without identifying the endpoint or capture time.

A spreadsheet may list users without showing whether it came from the authoritative identity system.

A PDF may record approval without proving that the approved data matches the data used in policy evaluation.

The problem is not the file format.

The problem is the unsupported claim.

Data alone is not evidence.

Provenance makes it usable.

Observation and remediation need separate records

A team observes that logging is disabled.

Someone enables logging.

The row becomes green.

The report says the issue is resolved.

But several events occurred:

1. The original state was observed.

2. A remediation action was approved.

3. The configuration was changed.

4. The post-change state was checked.

5. The checklist status was updated.

These are not one evidence object.

The original observation supports the finding.

The change record supports the remediation action.

The later observation supports post-change verification.

The checklist coordinates the sequence.

If the original evidence is replaced by a clean post-remediation screenshot, the audit trail loses the failure it claims to have corrected.

For that boundary, see Read-Only Collection as an Audit Boundary.

Evidence states must remain separate

A binary checklist encourages pass or fail.

The evidence pipeline needs more precise states.

control_fail means valid evidence was collected, but the observed state did not satisfy the policy.

invalid_evidence means evidence was supplied, but its integrity, provenance, identity, or linkage cannot be relied upon.

insufficient_evidence means required evidence is missing, incomplete, stale, or outside the intended scope.

These states are not interchangeable.

Missing evidence does not prove control failure.

A failed hash does not prove non-compliance.

A green checklist row does not repair either problem.

The useful chain

Control expectation
→ Checklist item
→ Evidence requirement
→ Evidence object
→ Validation result
→ Policy result
→ Audit narrative

The checklist coordinates the work.

The evidence object preserves the observation.

Validation determines whether the evidence is usable.

Policy evaluation tests a defined condition.

The report explains the result.

For the later verification test, see Can Your Audit Evidence Survive Replay?.

The position

A checklist can show that work was assigned, reviewed, attested, or closed.

It cannot by itself establish the underlying technical state.

It cannot prove complete scope.

It cannot prove continuous operation.

It cannot prove that the attached artifact was the object used for policy evaluation.

Checklist is coordination.

Evidence is proof material.

Policy evaluation is judgment.

Report is narrative.

Do not confuse them.

Related Reading

• Compliance Automation Starts at Evidence

• Read-Only Collection as an Audit Boundary

• Can Your Audit Evidence Survive Replay?

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and public discussion of this material.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This material is not legal, regulatory, audit, certification, implementation, or compliance advice.

It may contain screenshots, exported reports, configuration files, approval notes, and signed control narratives. A reviewer may be able to read it. A manager may be able to approve it. A report may already have been produced from it.

That does not mean the evidence can survive replay.

Replayable audit evidence is evidence that a later reviewer can identify, verify, re-evaluate, and compare with the original audit result without relying on the original operator’s memory.

Replay is not opening the same report again.

Replay is not generating a new export from the current system.

Replay is not running the latest policy against the latest data.

A replay must preserve the evidence and the context that produced the original decision.

The failure starts before the report. It starts when nobody can establish when the evidence was collected, where it came from, whether it changed after collection, or whether the policy result was produced from the same evidence object shown in the audit pack.

That is not a reporting problem.

It is an evidence integrity problem.

Compliance automation is not about producing cleaner reports. It is about making evidence harder to fake, harder to mutate, and easier to replay.

The replay question

Take one evidence item from an audit pack.

Ask:

• Can you identify the exact evidence object used in the original evaluation?

• Can you establish when it was collected?

• Can you identify what collected it?

• Can you identify the source system and target scope?

• Can you verify that it has not changed since it was sealed?

• Can you identify the policy version that evaluated it?

• Can the policy result be traced back to this exact evidence object?

• Can the replayed result be compared with the original result?

If the answer is no, the evidence may still be useful documentation.

It may help a reviewer understand the environment.

It may support an audit narrative.

But it is not yet strong replayable evidence.

Documentation helps explain.

Replayable evidence preserves the path to the decision.

The evidence object comes first

The first structure in a compliance automation pipeline is not the report.

It is the evidence object.

A minimum evidence object should be:

• timestamped

• source-bound

• scope-bound

• collector-identified

• integrity-sealed

• verified before evaluation

• linked to a specific policy result

Without these properties, the later audit narrative is built on unstable ground.

This is where many compliance automation efforts start too late. They begin with dashboards, templates, report generators, and control mappings.

Those may be useful.

They sit downstream.

If the evidence object cannot be trusted, the report only makes an untrusted object easier to read.

That is formatting, not assurance.

For the broader evidence-object model, see Compliance Automation Starts at Evidence.

A hash proves integrity, not truth

A cryptographic hash can help establish that a preserved file has not changed since it was sealed.

That boundary must remain explicit.

A valid hash does not prove that:

• the collector observed the correct target;

• the collection covered the required scope;

• the collector operated correctly;

• the source system was authoritative;

• or the evidence was sufficient for the control being evaluated.

A perfectly sealed object can still be incomplete, stale, incorrectly scoped, or collected from the wrong source.

Integrity is necessary.

Integrity is not sufficiency.

Replay also requires policy context

Preserving the evidence object is not enough.

The original policy context must also survive.

If the policy logic, supporting data, or input schema has changed, a later evaluation may still be valid. But it is a new evaluation, not a replay of the original decision.

At minimum, the audit record should identify:

• the policy;

• the policy version;

• the evidence object evaluated;

• the original result;

• and the time of evaluation.

Where supporting policy data affects the outcome, that context must also be preserved or referenced.

OPA should not evaluate raw trust.

It should evaluate verified evidence.

The result should remain traceable to the evidence and policy context that produced it.

Do not collapse different failure states

If the evidence hash fails, the correct result is not automatically non_compliant.

A failed integrity check means the evidence cannot safely support the evaluation.

That is an evidence-state failure.

At minimum, distinguish three outcomes:

control_fail

The evidence is usable, but the observed system state does not satisfy the defined control condition.

invalid_evidence

The evidence failed integrity, schema, provenance, or other admissibility checks.

The control should not be evaluated as though the evidence were valid.

replay_mismatch

The intended evidence and policy context were replayed, but the result differed from the original recorded result.

That indicates an inconsistency in the evaluation path.

It is not automatically a control failure.

A failed control says the observed state did not meet the expected condition.

Invalid evidence says the audit cannot safely determine whether the condition was met.

A replay mismatch says the decision path cannot be reproduced consistently.

Do not mix them.

MAS TRM-inspired, not MAS TRM-prescribed

MAS TRM-inspired compliance automation should not be treated as a claim that MAS prescribes this implementation.

It is an engineering interpretation of supervisory expectations.

The useful question is not:

Can we generate a report that sounds aligned?

The useful question is:

Can we produce evidence that can be collected, identified, sealed, verified, evaluated, and replayed?

A MAS TRM-inspired evidence pipeline should keep four layers separate:

1. Collection captures a bounded system state.

2. Evidence validation checks integrity, structure, and provenance.

3. Policy evaluation tests verified evidence against defined conditions.

4. The audit narrative explains the result and its limitations.

If these layers collapse into one report, the audit surface becomes fragile.

For the checklist boundary, see What a MAS TRM Checklist Cannot Prove.

A short self-test

Before treating an audit pack as automation-ready, test one evidence object.

Ask:

• Does it include a collection timestamp?

• Does it identify the source system and target scope?

• Does it identify the collector?

• Is there an integrity hash or equivalent seal?

• Was integrity verified before policy evaluation?

• Is the policy version identifiable?

• Is the result tied to this exact evidence object?

• Can the original and replayed results be compared?

This is not a remediation checklist.

It does not tell the operator how to fix the system.

It does not certify the control.

It tests whether the evidence can support a replayable audit conclusion.

Observation is not remediation.

Evidence is not a report.

A control is not proof.

A hash is not truth.

A rerun is not necessarily a replay.

The audit problem starts earlier.

If the evidence cannot survive replay, the report should not pretend to be stronger than the evidence beneath it.

Reports persuade.

Evidence survives.

Continue the evidence path

Replay only works if the surrounding evidence pipeline preserves its boundaries.

• Compliance Automation Starts at Evidence

• Read-Only Collection as an Audit Boundary

• What a MAS TRM Checklist Cannot Prove

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and public discussion of this material.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This material is not legal, regulatory, audit, certification, implementation, or compliance advice.

AI vendor risk assessment is not mainly a questionnaire problem.

It is an evidence conversion problem.

Most teams already have vendor questionnaires. They ask about security, privacy, subprocessors, SOC 2, access control, incident response, data retention, and contractual terms.

That is useful.

But it is not enough.

A vendor can answer every question and still leave the buyer without reviewable evidence.

The harder question is not:

Did the vendor answer?

The harder question is:

Can the buyer rely on this answer later?

That is the purpose of AI Vendor Evidence Gap Pack.

It helps buyers and reviewers turn AI vendor claims into evidence requests, weak-answer patterns, buyer questions, review notes, and usage boundaries.

It does not approve vendors.

It shows where the vendor’s claim stops and where the buyer still lacks evidence.

Problem

A vendor may say:

We do not train on your data.

That may be true.

But a review should not stop there.

The buyer still needs to know what “training” means. Does it include fine-tuning, evaluation, abuse monitoring, support review, retained logs, human review, embeddings, metadata, subprocessors, or model providers?

The buyer also needs to know where the claim appears.

A DPA clause is not the same as a marketing page.

A SOC 2 report is not automatically proof that the specific AI feature, model path, data flow, or product tier is covered.

A trust center can provide useful source material, but it is not a completed risk assessment.

The problem is not that every vendor claim is false.

The problem is that many vendor claims remain narrative claims.

They sound reassuring. They may be directionally useful. But they do not automatically become evidence requests, evidence gaps, buyer questions, review notes, or residual risk language.

That creates a gap in the review file.

If the vendor is approved or renewed, the buyer may still be unable to explain why the answer was accepted.

What this pack does

AI Vendor Evidence Gap Pack converts vendor-controlled language into reviewable structure:

AI vendor claim -> evidence source -> evidence gap -> buyer question -> usage boundary

This is not a longer questionnaire.

A questionnaire collects answers.

This pack helps review whether those answers are evidence-complete.

For each claim, the review asks:

• What exactly is being claimed?

• Where is the claim written?

• Is the source contractual, audited, official, public, marketing, or unsupported?

• Does it cover the actual product, plan, region, and use case?

• Does it cover the relevant data path?

• What is missing?

• What should the buyer ask next?

• What usage boundary is reasonable until the missing evidence is resolved?

The output is not a pass or fail conclusion.

The output is evidence structure.

Example

Vendor claim:

We do not train on your data.

Why it sounds sufficient:

It appears to answer the main AI privacy concern.

What it actually proves:

It may support a narrow commitment about model training, depending on the source, scope, product, plan, contract, and date.

What it does not prove:

It does not prove prompt logging, output storage, file retention, metadata handling, embeddings, abuse monitoring, support access, human review, subprocessors, deletion controls, retention period, data residency, audit logs, or exportability.

Weak-answer pattern:

The vendor repeats that it does not train on customer data, but does not define training, retention, logging, evaluation, human review, support access, subprocessors, or exceptions.

Evidence request:

Provide the data flow for prompts, outputs, uploaded files, logs, embeddings, metadata, moderation events, support access, subprocessors, model providers, and human review queues. Include retention period, access roles, deletion controls, tenant settings, contractual commitments, and documented exceptions.

Review note:

The claim is directionally useful but not evidence-complete. It addresses model training use, but does not establish how operational data is handled after inference.

Usage boundary:

Do not rely on this claim for customer data, confidential data, regulated data, proprietary source code, or production approval until retention, logging, human review, subprocessor, and deletion boundaries are evidenced.

The point is not to accuse the vendor.

The point is to avoid accepting a claim before it becomes reviewable evidence.

Who it is for

This pack is for people involved in AI vendor review before procurement, renewal, internal approval, or audit preparation.

That includes security reviewers, compliance teams, procurement operators, privacy leads, IT managers, founders, operators, and small teams adopting AI tools.

It is especially useful for teams that do not yet have a mature AI vendor review process, but still need to ask defensible questions before real data use.

The review should not start with the vendor name alone.

The real review unit is:

vendor + product + plan + use case + data type + region + contract terms + evidence date

The same vendor may be acceptable for public marketing drafts and not acceptable for customer records, source code, employee data, regulated data, or automated decisioning.

Context matters.

What it is not

AI Vendor Evidence Gap Pack is not certification, vendor rating, legal advice, regulatory advice, audit opinion, procurement approval, compliance guarantee, or a pass/fail conclusion.

It is not a substitute for legal, security, privacy, audit, procurement, or business owner review.

It is also not a generic AI governance checklist.

The product boundary is simple:

Reading path

Each claim teardown follows the same review path:

vendor claim → evidence source → evidence gap → buyer question → usage boundary

Start with the core frame:

1. AI Vendor Risk Is Not a Questionnaire Problem

Then read the claim teardowns:

2. Vendor Says It Does Not Train on Your Data. What Evidence Should You Ask For?

3. A Trust Center Is Not an AI Vendor Risk Assessment

4. Why SOC 2 Does Not Prove the AI Vendor Data Path Is Covered

5. Human Review Is a Data Exposure Path Unless It Is Bounded

6. Subprocessor Lists Do Not Show the Actual AI Data Path

Boundary

This material is for evidence structuring and review preparation.

It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.

I now have a sanitized sample evidence gap memo showing how this looks in a review file.

Reply if you want the sample.

That is the weakness.

The screenshot is taken after the audit request arrives. The log export is prepared after the system has moved on. The control owner writes how the process is supposed to work.

The report becomes coherent.

Coherence is not proof.

Compliance evidence automation is not template filling. It means collecting bounded observations from source systems, preserving provenance, validating integrity, evaluating policy, and producing a reviewable evidence package.

A clean audit pack can still be built on weak evidence.

That is the problem compliance automation must solve first.

The report is too late

A report is downstream.

It cannot rescue stale evidence.

It cannot repair a missing timestamp.

It cannot establish which source system was observed.

It cannot prove that the evidence remained unchanged after collection.

Most automation starts at the wrong layer.

It generates summaries.

It fills templates.

It writes control narratives.

That saves labour.

It does not harden evidence.

Evidence needs provenance

Data alone is not evidence.

An evidence object should tell us:

• what was observed;

• when it was observed;

• where it came from;

• what collected it;

• whether it changed after collection;

• and which policy result was produced from it.

Without that structure, the reviewer may have data but not a defensible audit trail.

A file name is not provenance.

A screenshot without capture context is not a complete evidence object.

A manually entered timestamp is only an assertion unless the collection process supports it.

An evidence object becomes replayable only when its identity, provenance, integrity state, and policy context can be reconstructed later.

For the replay test, see Can Your Audit Evidence Survive Replay?.

Collection is not correction

The collector should not fix the system.

If collection changes the target, the failed state may disappear before it is recorded.

Operations may like that.

Audit should not.

Read-only collection is not a tooling preference.

It is restraint.

Pull the configuration.

Timestamp it.

Seal it.

Evaluate it.

Remediate later.

Observation and remediation are different jobs.

Read-only does not prove that collection was complete or correct. It preserves a narrower boundary: the collector is not intended to correct the state it is supposed to observe.

For that boundary, see Read-Only Collection as an Audit Boundary.

Integrity comes before judgment

A hash does not prove compliance.

It does not prove that the collector observed the correct system.

It does not prove that the required scope was complete.

It proves a narrower claim:

The preserved bytes match the bytes that were sealed.

That is useful.

If integrity verification fails, the audit path should stop.

OPA should not evaluate evidence that has already failed its own integrity boundary.

The correct result is not automatically non_compliant.

It is invalid_evidence.

Bad evidence should not produce a clean result.

OPA is not the system

Open Policy Agent is a policy evaluator.

Not a collector.

Not a repair tool.

Not an audit writer.

Its job is narrow: take validated evidence and evaluate whether the observed state satisfies a defined policy condition.

That narrowness is the point.

The collector observes.

The validator checks evidence.

OPA evaluates policy.

The audit narrative explains the result.

Policy should stay deterministic.

Narrative comes later.

TLS shows the boundary

Take an Apache HTTPS service.

The weak audit asks whether HTTPS is enabled.

The stronger audit asks:

• when the certificate was observed;

• which endpoint was inspected;

• when the certificate expires;

• which signature algorithm was observed;

• whether the evidence was sealed;

• and whether it was verified before policy evaluation.

The first approach checks a claim.

The second preserves evidence.

A certificate expiring within 48 hours is not automatically a universal compliance failure.

It becomes a control result only when the applicable policy defines that threshold and the evidence is valid for the intended scope.

The audit narrative should stay tied to that boundary:

No legal conclusion.

No claim about the entire control environment.

Just evidence, condition, and result.

MAS TRM-inspired means engineering interpretation

MAS TRM is the context here.

It does not prescribe this schema, collector, OPA rule, or evidence pipeline.

MAS TRM-inspired means engineering interpretation.

The engineering task is not to rewrite regulatory language into softer prose.

It is to turn a control expectation into an inspectable evidence structure.

A control expectation needs:

• a defined observation;

• an evidence schema;

• a collector;

• a timestamp;

• an integrity check;

• a policy condition;

• and an audit narrative.

The schema is not the control.

The policy result is not the entire audit conclusion.

That is the machine boundary.

The position

Compliance automation is not report automation.

Report automation makes audit packs faster.

Evidence automation makes control claims harder to fake, harder to mutate, and easier to replay.

That is the useful distinction.

Reports persuade.

Evidence survives.

Related Reading

• Can Your Audit Evidence Survive Replay?

• Read-Only Collection as an Audit Boundary

Origin

CodeYourCompliance

Website: https://www.codeyourcompliance.com/

GitHub: https://github.com/codeyourcompliance

Attribution is requested for forks, references, adaptations, and discussions.

Scope Boundary

MAS TRM-inspired means engineering interpretation.

This is not legal, regulatory, audit, certification, compliance, or implementation advice.