Can Your Audit Evidence Survive Replay?
A short test for timestamped, sealed, and policy-evaluable compliance evidence.
An audit pack can look complete and still fail as evidence.
It may contain screenshots, exported reports, configuration files, approval notes, and signed control narratives. A reviewer may be able to read it. A manager may be able to approve it. A report may already have been produced from it.
That does not mean the evidence can survive replay.
Replayable audit evidence is evidence that a later reviewer can identify, verify, re-evaluate, and compare with the original audit result without relying on the original operator’s memory.
Replay is not opening the same report again.
Replay is not generating a new export from the current system.
Replay is not running the latest policy against the latest data.
A replay must preserve the evidence and the context that produced the original decision.
The failure starts before the report. It starts when nobody can establish when the evidence was collected, where it came from, whether it changed after collection, or whether the policy result was produced from the same evidence object shown in the audit pack.
That is not a reporting problem.
It is an evidence integrity problem.
Compliance automation is not about producing cleaner reports. It is about making evidence harder to fake, harder to mutate, and easier to replay.
A report can persuade.
Evidence must survive.
The replay question
Take one evidence item from an audit pack.
Ask:
Can you identify the exact evidence object used in the original evaluation?
Can you establish when it was collected?
Can you identify what collected it?
Can you identify the source system and target scope?
Can you verify that it has not changed since it was sealed?
Can you identify the policy version that evaluated it?
Can the policy result be traced back to this exact evidence object?
Can the replayed result be compared with the original result?
If the answer is no, the evidence may still be useful documentation.
It may help a reviewer understand the environment.
It may support an audit narrative.
But it is not yet strong replayable evidence.
Documentation helps explain.
Replayable evidence preserves the path to the decision.
The evidence object comes first
The first structure in a compliance automation pipeline is not the report.
It is the evidence object.
A minimum evidence object should be:
timestamped
source-bound
scope-bound
collector-identified
integrity-sealed
verified before evaluation
linked to a specific policy result
Without these properties, the later audit narrative is built on unstable ground.
This is where many compliance automation efforts start too late. They begin with dashboards, templates, report generators, and control mappings.
Those may be useful.
They sit downstream.
If the evidence object cannot be trusted, the report only makes an untrusted object easier to read.
That is formatting, not assurance.
For the broader evidence-object model, see Compliance Automation Starts at Evidence.
A hash proves integrity, not truth
A cryptographic hash can help establish that a preserved file has not changed since it was sealed.
That boundary must remain explicit.
A valid hash does not prove that:
the collector observed the correct target;
the collection covered the required scope;
the collector operated correctly;
the source system was authoritative;
or the evidence was sufficient for the control being evaluated.
A perfectly sealed object can still be incomplete, stale, incorrectly scoped, or collected from the wrong source.
Integrity is necessary.
Integrity is not sufficiency.
Replay also requires policy context
Preserving the evidence object is not enough.
The original policy context must also survive.
If the policy logic, supporting data, or input schema has changed, a later evaluation may still be valid. But it is a new evaluation, not a replay of the original decision.
At minimum, the audit record should identify:
the policy;
the policy version;
the evidence object evaluated;
the original result;
and the time of evaluation.
Where supporting policy data affects the outcome, that context must also be preserved or referenced.
OPA should not evaluate raw trust.
It should evaluate verified evidence.
The result should remain traceable to the evidence and policy context that produced it.
Do not collapse different failure states
If the evidence hash fails, the correct result is not automatically non_compliant.
A failed integrity check means the evidence cannot safely support the evaluation.
That is an evidence-state failure.
At minimum, distinguish three outcomes:
control_fail
The evidence is usable, but the observed system state does not satisfy the defined control condition.
invalid_evidence
The evidence failed integrity, schema, provenance, or other admissibility checks.
The control should not be evaluated as though the evidence were valid.
replay_mismatch
The intended evidence and policy context were replayed, but the result differed from the original recorded result.
That indicates an inconsistency in the evaluation path.
It is not automatically a control failure.
A failed control says the observed state did not meet the expected condition.
Invalid evidence says the audit cannot safely determine whether the condition was met.
A replay mismatch says the decision path cannot be reproduced consistently.
Do not mix them.
MAS TRM-inspired, not MAS TRM-prescribed
MAS TRM-inspired compliance automation should not be treated as a claim that MAS prescribes this implementation.
It is an engineering interpretation of supervisory expectations.
The useful question is not:
Can we generate a report that sounds aligned?
The useful question is:
Can we produce evidence that can be collected, identified, sealed, verified, evaluated, and replayed?
A MAS TRM-inspired evidence pipeline should keep four layers separate:
Collection captures a bounded system state.
Evidence validation checks integrity, structure, and provenance.
Policy evaluation tests verified evidence against defined conditions.
The audit narrative explains the result and its limitations.
If these layers collapse into one report, the audit surface becomes fragile.
For the checklist boundary, see What a MAS TRM Checklist Cannot Prove.
A short self-test
Before treating an audit pack as automation-ready, test one evidence object.
Ask:
Does it include a collection timestamp?
Does it identify the source system and target scope?
Does it identify the collector?
Is there an integrity hash or equivalent seal?
Was integrity verified before policy evaluation?
Is the policy version identifiable?
Is the result tied to this exact evidence object?
Can the original and replayed results be compared?
This is not a remediation checklist.
It does not tell the operator how to fix the system.
It does not certify the control.
It tests whether the evidence can support a replayable audit conclusion.
Observation is not remediation.
Evidence is not a report.
A control is not proof.
A hash is not truth.
A rerun is not necessarily a replay.
The audit problem starts earlier.
If the evidence cannot survive replay, the report should not pretend to be stronger than the evidence beneath it.
Reports persuade.
Evidence survives.
Continue the evidence path
Replay only works if the surrounding evidence pipeline preserves its boundaries.
Origin
CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance
Attribution is requested for forks, references, adaptations, and public discussion of this material.
Scope Boundary
MAS TRM-inspired means engineering interpretation.
This material is not legal, regulatory, audit, certification, implementation, or compliance advice.