An evidence package can contain every required file and still fail validation.
That happened while testing the CodeYourCompliance TLS replay reference package on Windows.
The first 20 unit tests passed. But when the validator checked the actual repository files, it returned:
closure_status = blocked
8 x sha256_mismatchAll eight files were present. None matched the SHA256 values recorded in the manifest.
This was a synthetic TLS example, not a production incident. More importantly, validation failed before policy evaluation. There was no basis for reporting a TLS control failure.
The files were present. The bytes were different.
The problem came from Git line-ending conversion.
Git’s index stored LF files. The Windows working tree contained CRLF versions.
i/lf w/crlf attr/text eol=lfThe JSON data had not changed in meaning. Neither had the Rego policy logic. But SHA256 checks bytes, not parsed content.
Git documents this behavior in its gitattributes reference.
Our initial tests used temporary synthetic files. They did not cover what happened after Git checkout.
Adding a regression test against the actual repository files exposed the problem: 20 tests passed, and the new test failed.
The validator was correctly rejecting the package.
Why we kept the original hashes
We could have recalculated the SHA256 values on Windows and updated the manifest.
That would have made validation pass, but it would also have replaced the reference we intended to verify.
Instead, we retrieved the original bytes from Git’s index, checked them against the manifest, and confirmed that the working-tree differences were limited to line endings.
Only then did we restore the files.
Two earlier Git commands had returned success without correcting the bytes. Their exit codes were not sufficient evidence that the repair had worked.
The final Windows results were:
8/8 SHA256 matched
21/21 tests passed
closure_status = satisfied
replay_executed = false
decision_executed = false
control_status = unknownThe procedure is recorded in the Windows LF Repair document.
A manifest cannot define its own completeness
The Windows failure concerned file integrity. Package closure addresses another question.
Can we still obtain the inputs needed to reproduce the original assessment?
As discussed in Can Your Audit Evidence Survive Replay?, retaining an evidence file is not the same as preserving the conditions under which it was evaluated.
For the TLS example, those dependencies include the evidence digest, schemas, evaluation context, policy, replay implementation, and runtime requirements.
Consider a certificate with 35 days remaining. It may pass a 30-day threshold and fail a 45-day threshold. Replaying the evidence with today’s threshold would not reproduce the earlier decision.
A manifest can also omit a required dependency. Checking only the files listed in that manifest would not detect the omission.
Our Evidence Package Closure Validator therefore uses a fixed tls-replay-v0.1 profile defining eight required roles and paths.
The manifest cannot declare itself complete simply by leaving out a required file.
But the fixed profile has limits.
It defines what this validator expects. It does not independently establish that the manifest is authentic, that the profile has been approved by a trusted authority, or that every historical runtime dependency has been preserved.
Someone able to replace both the files and the manifest could create a new, internally consistent package.
The Evidence Package Closure Contract records these boundaries explicitly.
Closure is not replay
The current implementation checks local file availability, expected paths, SHA256 correspondence, and the evidence-digest binding.
It does not authenticate the source, reconstruct the historical execution environment, or execute OPA.
This distinction also applies to evidence revisions. In Corrected Evidence Must Not Erase the Original, we examined why corrections must preserve the inputs used by historical decisions.
The same constraint applies here. A newer file or a matching manifest cannot substitute for the original assessment inputs.
Collection records observations. Evidence validation checks retained material. Policy evaluation produces a control decision. Remediation changes the target system.
The Windows repair changed only local synthetic reference files. It was not remediation of a TLS control failure.
The completed tests establish that this particular package satisfies its defined local closure checks. They do not establish that the historical assessment has been replayed.
A replayable assessment needs more than the original evidence file. It needs the exact inputs that produced the decision, and a way to verify them before execution.
Public reference artifacts
The accompanying technical material is available in the CodeYourCompliance evidence-validation-pipeline repository:
Origin and scope
CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance
CodeYourCompliance explores replayable evidence, read-only evidence collection, provenance, integrity verification, and policy-as-code.
Attribution is requested for forks, references, adaptations, and technical discussions.
This work uses synthetic evidence and a bounded TLS reference implementation.
MAS TRM-inspired means engineering interpretation, not a prescribed regulatory implementation. This material is not legal, regulatory, audit, or certification advice.