A reviewer examines a TLS assessment recorded several months earlier. The original decision was PASS. The assessment is executed again, and the new result is also PASS.
But were both decisions produced using the same evidence, policy, evaluation context, and execution logic?
Two identical results cannot answer that question. Different assessment conditions may produce the same outcome, making a successful rerun appear more conclusive than it actually is.
Package closure was only the first step
Our earlier synthetic TLS experiment exposed a Windows checkout problem. All eight required files were present, but Git line-ending conversion changed their working-tree bytes. The files failed SHA256 verification against the manifest, although their JSON and Rego content remained semantically unchanged. Git documents this behavior in its gitattributes reference.
After restoring the expected bytes, the Evidence Package Closure Validator returned closure_status = satisfied.
That confirmed the package met its fixed file, path, and hash requirements. It did not execute a control decision or authenticate a historical assessment.
As discussed in Can Your Audit Evidence Survive Replay?, retaining evidence is different from preserving the conditions needed to evaluate it again.
We therefore added Replay Comparison Receipt v0.1, which executes the existing TLS evaluator and compares its output against a recorded reference.
The reference is explicitly synthetic and unattested. It is not a production assessment record.
Why comparing PASS with PASS is insufficient
Consider a TLS certificate with 35 days remaining. Under a 30-day minimum-validity threshold, it passes. Under a 45-day threshold, the same certificate fails.
The certificate has not changed. The evaluation context has.
Conversely, two assessments may both return PASS despite using different policy versions or thresholds. Comparing only control_status would miss those differences.
Our comparator first invokes package closure validation, then checks all eight artifact hashes against the recorded baseline. These bindings cover the evidence, schemas, evaluation context, policy, replay implementation, and other required inputs.
Only after the references and selected decision engine agree does it invoke the existing replay.py.
The comparator then checks the hashes reported by the runner, implementation identity and version, execution path, derived facts, and selected decision fields.
A new PASS with a different policy digest cannot return matched.
This does not establish policy correctness. It establishes whether the new execution corresponds to the supplied reference within the defined comparison scope.
The comparator also produced an unexpected problem
During the review of PR #16, we discovered that Python’s ordinary equality operator could conceal a JSON type difference:
False == 0.0 # TrueA Boolean false and a numeric 0.0 are different JSON values, but Python considers them equal. When comparing nested assessment results, this could incorrectly produce matched.
We replaced the comparison with a recursive check that preserves both value and type distinctions, then added two regression tests covering Boolean, integer, and floating-point differences.
The problem was not in the TLS control. It was in the logic used to determine whether two assessments agreed.
This resembled the earlier Windows issue. In both cases, a seemingly ordinary implementation detail affected the validity of the verification result.
What the Windows tests established
After the correction, all 18 unit tests passed on Windows. We then ran the comparator against the TLS package actually checked into the repository, using the synthetic recorded baseline and the existing builtin evaluator rather than a simulated runner.
The relevant output was:
closure_status = satisfied
comparison_status = matched
runner_invoked = true
decision_executed = true
recorded_result = pass
replayed_result = pass
mismatched_fields = []The command exited with code 0, and the tracked Git working tree remained clean. The execution details and review correction are preserved in the merged PR #16.
The comparator also distinguishes input_mismatch, decision_mismatch, and blocked. These states describe problems with assessment inputs, comparison results, or execution. They do not automatically indicate that the target system failed a TLS control.
A matching result still needs a trustworthy reference
The receipt retained several explicit limitations:
policy_artifact_executed = false
baseline_authentication = not_assessed
manifest_authentication = not_assessed
runtime_equivalence = not_assessed
historical_replay_verified = falseThis run used the builtin evaluator, so the Rego policy was not executed through OPA. The comparator also did not authenticate the recorded baseline or prove that the historical runtime environment had been reproduced.
Someone able to replace the evidence, manifest, and baseline together could construct a new, internally consistent package. Its hashes could match, its assessment could return PASS, and the comparator could report matched.
That would establish consistency against the supplied reference, not the authenticity of the original assessment.
Collection records observations. Evidence validation checks retained inputs. Policy evaluation produces a decision. Replay comparison checks a new execution against a recorded reference. Remediation, which changes the target system, remains outside this read-only workflow.
Our Windows tests established that the current TLS execution matched the supplied synthetic assessment on the checked fields and bindings. Establishing historical replay requires something further: an independently verifiable baseline that cannot be silently replaced alongside the evidence being examined.
Origin and scope
CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance
CodeYourCompliance explores replayable evidence, read-only evidence collection, provenance, integrity verification, and policy-as-code.
Attribution is requested for forks, references, adaptations, and technical discussions.
MAS TRM-inspired means engineering interpretation, not legal, regulatory, audit, certification, or compliance advice
Public reference artifacts
The accompanying technical material is available in the CodeYourCompliance evidence-validation-pipeline repository: