A control passes in one assessment and fails in the next.
The usual explanation is that something changed in the target system. That is possible, but it is not the only explanation. The evidence may be unchanged. The normalized facts may be unchanged. The evaluation context may also be unchanged.
The policy may have changed instead.
If the assessment history preserves only:
previous result: PASS
current result: FAIL
then the difference is visible, but its cause is not.
That becomes an evidence problem before it becomes a control problem.
The result depends on the policy that produced it
Policy-as-code makes evaluation repeatable. It does not make the result independent of the rule being evaluated.
The synthetic example in the public Policy Evaluation Provenance Example keeps one policy input constant:
days_to_expiry = 30
minimum_certificate_days_remaining = 30
Policy version 0.1.0 evaluates:
days_to_expiry >= minimum_certificate_days_remaining
The modeled result is PASS.
Policy version 0.2.0 evaluates:
days_to_expiry > minimum_certificate_days_remaining
The modeled result is FAIL.
The target-state identifier, normalized fact and evaluation context are held constant by construction. Only the policy semantics change.
The example does not prove that a real target remained unchanged between two assessments. It isolates one variable so that the decision difference can be attributed to the policy change inside the synthetic scenario.
The useful boundary is therefore narrower:
same policy input
+ different policy semantics
-> different decision
result difference alone
does not establish target-state change
A changed result needs more context before it can be called drift.
Policy version is only one part of evaluation provenance
Recording a field such as:
policy_version = 0.2.0
is useful, but replay needs more than a version label.
The Policy Evaluation Provenance Contract models the decision path with fields such as:
evaluation:
engine:
engine_version:
policy_id:
policy_version:
rule_id:
rule_version:
policy_artifact_ref:
policy_artifact_sha256:
policy_input_ref:
policy_input_sha256:
evaluation_context_ref:
evaluation_context_sha256:
implementation_version:
evaluated_at_utc:
decision_executed:
result:
The point is not to collect metadata for its own sake.
A reviewer needs to know which policy artifact was evaluated, which input it consumed, which context was applied, which decision entry point ran, and which engine and surrounding implementation produced the result.
Without that information, a historical assessment can easily be replayed using today’s policy instead of the policy that produced the original decision.
The evidence may be the same. The replay is not.
A policy hash does not establish policy authority
The policy artifact can also be hashed.
That improves identity. A SHA256 digest can show which policy bytes were referenced by an evaluation record and whether the preserved artifact later changed.
It does not show whether those bytes should have been used.
This distinction matters:
policy byte identity
!=
policy release authority
A digest does not prove that the policy was approved, current at the time of evaluation, applicable to the assessed scope, or released by an authorised control owner.
A replay package can therefore reproduce a decision against the wrong policy artifact with perfect integrity.
The replay is technically faithful. The policy selection may still be wrong.
Signed policy bundles, approval records and trusted policy-release provenance are separate concerns.
Deterministic evaluation does not validate the policy
OPA can evaluate the same input against the same Rego artifact and return the same result repeatedly.
That proves something useful: the identified evaluation path is deterministic.
It does not prove that the policy represents the correct control interpretation.
A threshold can be wrong. An exception may be missing. A rule may be valid for one system scope and unsuitable for another.
Evaluation provenance answers:
What policy path produced this result?
It does not answer:
Was this the correct policy for this control?
That judgment remains outside the policy engine.
Replay has to preserve the historical decision path
This repository already contains a bounded Replayable TLS Control.
Its pipeline is:
evidence schema
-> context schema
-> integrity
-> freshness
-> derived facts
-> policy evaluation
-> assessment result
That reference implementation records hashes for evidence, schemas, evaluation context and policy, together with implementation and runtime provenance.
The new policy-evaluation provenance artifact does not replace that implementation. It extracts one specific rule from it:
A historical control result should remain tied to the policy and evaluation path that produced it.
That makes this problem slightly different from the earlier question of whether a control result is replayable at all.
The first question is:
Can I reproduce this result?
The next question is:
Am I reproducing the historical decision,
or re-evaluating historical evidence
under a different policy?
Both operations may be useful.
They should not be confused.
The decision path has more than one moving part
Once collection, normalization and policy evaluation are separated, a changed assessment result can have several explanations:
target changed
collector changed
transformation changed
policy input changed
evaluation context changed
policy changed
engine or implementation changed
A PASS -> FAIL transition does not identify which one occurred.
The control result is the end of the pipeline. It is not a complete explanation of the pipeline.
Public reference artifacts
The implementation references are available in the public CodeYourCompliance evidence-validation-pipeline:
The synthetic comparison does not claim to be an OPA runtime capture. The repository also does not yet implement signed policy bundles, trusted policy-release attestation or generic cross-policy regression testing.
Those remain separate implementation steps.
A control result is downstream of evidence, context and policy.
If the policy path disappears, part of the decision history disappears with it.
Origin and scope
CodeYourCompliance
Website: https://www.codeyourcompliance.com/
GitHub: https://github.com/codeyourcompliance
Attribution is requested for forks, references, adaptations and technical discussions.
MAS TRM-inspired means engineering interpretation. This project does not provide legal, regulatory, audit, certification or compliance advice.