A Control Result Is Not Replayable Evidence
Why PASS/FAIL cannot show whether the system changed, the decision conditions changed, or the original conclusion can still be replayed.
A host passed a control yesterday and failed it today. The retained record shows only this:
2026-07-29: PASS
2026-07-30: FAILThe first question is whether the host changed or the test changed. The record cannot answer it.
The package inventory may have moved, but the baseline, freshness rule, policy or evaluator may also have changed. The same evidence can produce a different result when it is judged under a different evaluation context.
With only PASS and FAIL, there is no way to separate a change in the system from a change in the decision conditions. The conclusions were retained. The basis for them was not.
That is why a stored control result is not replayable evidence. An earlier note asked whether audit evidence can survive replay; the harder implementation question is what must be retained for that replay to mean anything.
What must survive the original decision
Replayable evidence is not the same as retained documentation. Replay does not mean running the same scanner again.
A new scan observes the system as it exists now. Replay reconstructs a previous decision from the artifacts that existed when that decision was made.
For that reconstruction to be credible, the retained package must include more than the observation. It must also preserve the schema that admitted the evidence, the context that supplied the threshold or baseline, the policy artifact, the selected decision engine, the implementation version and an integrity digest binding the assessment to the original evidence bytes.
Each artifact answers a different review question.
The evidence shows what the collector observed. The schema shows what input was considered admissible. The context records the rule parameters used at that time. The policy contains the decision logic. Runtime metadata identifies the execution path. The digest shows whether the retained evidence has changed since evaluation.
If any of these elements is missing, part of the decision must be inferred rather than replayed.
A timestamped result may still be useful operationally, but it is only a historical claim unless the decision path can be reconstructed.
The collector should not decide compliance
Consider a read-only collector operating within an explicit audit boundary. It returns the following package inventory:
{
"packages": [
{"name": "openssl", "version": "3.0.13"},
{"name": "python3", "version": "3.12.3"},
{"name": "curl", "version": "8.5.0"}
]
}This tells us what the collector observed. It does not tell us whether the host complies with a package baseline.
That decision depends on a separate context:
{
"required_packages": [
{"name": "openssl", "version": "3.0.13"},
{"name": "python3", "version": "3.12.3"},
{"name": "curl", "version": "8.5.0"}
],
"unexpected_package_mode": "ignore"
}If the required OpenSSL version changes, the same inventory may fail. If unexpected packages are changed from ignore to fail, an additional package may alter the result even though nothing on the host has changed.
The observation and the assessment therefore belong to different layers.
Package names and versions come from collection. Missing packages, mismatched versions and unexpected packages are derived by comparison. Pass or fail is produced only after policy evaluation.
Writing the control result back into the evidence object hides those transitions. The final document becomes easier to read, but harder to inspect. A reviewer can see the conclusion but cannot tell which facts came from the host, which were derived by the replay engine and which depended on policy choices.
Invalid, stale and failed are different states
A control should fail only after admissible evidence reaches the policy and the required condition is not met.
If the evidence does not satisfy its schema, the policy should not evaluate it. A missing timestamp, malformed package entry or incorrect field type is a problem with the evidence path, not proof that the host failed the control.
The assessment should record that distinction:
evidence_status = invalid_evidence
control_status = unknown
decision_executed = falseEvidence can remain intact and still become too stale to support a current conclusion. In this control, stale evidence creates a different problem from either invalid evidence or policy failure. A package inventory may be structurally valid and still be too old for the decision being attempted.
Suppose the context requires evidence collected within the previous 24 hours. An inventory from three days ago cannot support a current package-baseline decision. It does not prove that the host is non-compliant. It proves that the available observation is no longer fresh enough to answer the question.
evidence_status = stale_evidence
control_status = unknown
decision_executed = falseThese distinctions are not cosmetic. They imply different actions.
A failed control may require investigation of the host or baseline. Stale evidence requires recollection. Invalid evidence requires examination of the collector, schema or processing pipeline.
A dashboard that reduces all three conditions to red has removed information that operations and audit both need.
Collection and policy execution are separate boundaries
The earlier implementation note moved from evidence principles to an executable control. The public reference artifacts now apply that structure through a fixed replay sequence:
evidence schema gate
-> context schema gate
-> integrity gate
-> freshness gate
-> derived facts
-> policy evaluation
-> assessment output
For the package-baseline example, the comparison stage derives three groups of facts:
missing_packages
version_mismatches
unexpected_packages
The policy then decides how those facts affect the control result.
A missing required package causes failure. A version mismatch causes failure. An unexpected package may be ignored or treated as a violation depending on the explicit context.
The collector does not need to know any of this. Its job is to report the inventory accurately and preserve its collection metadata. Policy belongs later in the pipeline, where its input and execution can be inspected independently.
The same boundary applies to the decision engine.
A built-in evaluator can implement the same decision semantics as a Rego policy, but it does not execute the Rego artifact. That difference must appear in the assessment:
builtin:
decision_executed = true
policy_artifact.executed = false
opa:
decision_executed = true
policy_artifact.executed = trueWithout this metadata, a result may appear to have been produced by a policy artifact that was never run. The conclusion could be correct while its provenance is false.
Replay should stop before remediation
The package-baseline control can identify a missing package, an incorrect version or an unexpected installation. It does not install, remove or upgrade software.
That is a deliberate boundary.
Remediation requires broader permissions and produces a different class of evidence: authorization records, execution logs, change results and post-remediation observations. Combining those actions with collection makes it harder to establish what state existed before the tool intervened.
A read-only collector preserves the condition it was asked to observe. A separate remediation workflow may act on the assessment later, but it should produce its own evidence package.
Collection establishes what was seen. Correction changes what will be seen next.
Public reference artifacts
The replay structure described here is implemented in the public codeyourcompliance/evidence-validation-pipeline repository.
The current examples are:
The TLS example applies the pipeline to certificate expiry. The OS package example applies it to required packages, version mismatches and unexpected software.
Both packages retain the evidence, validation schemas, evaluation context, integrity information, policy artifact and execution metadata needed to inspect how an assessment result was produced. Their tests also show how the pipeline behaves when evidence is stale, malformed or modified after hashing.
These examples do not prove that the underlying systems are compliant. They show that a control decision can be preserved in a form that is more inspectable than a stored PASS or FAIL.
They also stop before remediation. Installation, removal, upgrade and other corrective actions belong to a separate workflow with separate permissions and evidence.
Replayability does not establish that a decision was correct. It establishes whether the basis of that decision can still be examined.
Origin and scope
CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance
Attribution is requested for forks, references, adaptations and technical discussions.
MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.