Quality and Evidence Architecture

3. Enduring Principle

Quality enables autonomy

Autonomy is safe only to the extent that the system can detect, contain, and explain failure. Quality engineering is therefore not a final gate after agent work. It is the control system that makes greater execution authority possible.

The factory should trust a claim only when it can connect the claim to frozen criteria, an exact artifact, an independent method, a known verifier, and a current result.

Separate criteria, artifacts, receipts, and decisions

These concepts must not be collapsed:

Acceptance criterion states what must be true. It includes an identifier, outcome, method, pass condition, required evidence, independence requirement, and waiver policy.

Artifact is something produced or examined: source diff, commit, binary, test output, screenshot, trace, coverage report, security finding, or deployment record. An artifact is not automatically evidence of a criterion.

Verification receipt records that a known verifier applied a defined method to an exact artifact in a defined environment and observed a result.

Quality gate evaluates whether the current set of receipts satisfies policy. It does not create the underlying evidence.

Acceptance decision is the accountable judgment that the governed outcome is acceptable. A passing gate can make the work eligible; it does not eliminate the decision owner.

Evidence envelope

A useful evidence receipt should contain:

• Mission, Plan, WorkOrder, criterion, Task, and Attempt identity;
• WorkOrder and Plan revision;
• repository, base commit, head SHA, branch, and artifact hash;
• verifier identity, role, method, command, and tool version;
• execution environment and relevant configuration digest;
• observed result and machine-readable status;
• creation time, validity window, and retention classification;
• source artifacts and stable locations;
• confidence or uncertainty when the method is probabilistic;
• waiver or exception linkage; and
• invalidation and supersession history.

The receipt should be immutable. A later result supersedes it rather than rewriting what happened.

Evidence states have precise meaning

Pass means the verifier observed the defined pass condition for the exact artifact and context.

Fail means the pass condition was not met.

Pending or unknown means the factory lacks a conclusive result. Unknown must never be converted to pass for convenience.

Stale means the result was once usable but no longer applies because its artifact, environment, workflow, policy, or validity window changed.

Waived means an accountable human accepted a scoped exception with a reason, expiry, and compensating control. Waived does not mean passed.

Not applicable means policy determined that a criterion does not apply to the current scope. It should remain distinguishable from missing evidence.

Evidence must be fresh and artifact-specific

Freshness is not only time. Evidence becomes stale when any relevant assumption changes:

• source or head SHA changes;
• affected files or dependencies change;
• WorkOrder or Plan revision changes;
• validation method or workflow changes;
• environment or configuration changes;
• approval or receipt validity expires;
• a reopen decision invalidates the criterion; or
• a newer contradictory result appears.

Selective invalidation is preferable to discarding everything. The system must prove which criteria are unaffected. When it cannot, it should invalidate conservatively.

Validation must be independent

The implementation worker cannot be the sole authority that declares its own success. Independent validation requires:

• a separate execution identity and path;
• frozen acceptance criteria defined before the result;
• a clean or controlled validation environment;
• exact artifact identity;
• independently executed commands or checks;
• immutable receipts; and
• no permission to modify the artifact under evaluation.

Using a different model can reduce correlated reasoning error, but model diversity alone does not establish independence. Two agents sharing the same state, commands, and assumptions can reproduce the same mistake.

Validators are not voters

Security, performance, correctness, and accessibility validators answer different questions. Two passes do not outvote a security failure. When valid receipts conflict, the conflict itself becomes evidence and governance increases.

The factory should create a Risk Review that identifies the conflicting claims, methods, artifacts, severity, freshness, and safe options. Random retry is not a resolution unless a new hypothesis explains why another run is meaningful.

Continuous validation

Validation should occur throughout the lifecycle:

1. intent validation checks that the outcome and criteria are testable;
2. plan validation checks scope, dependencies, risk, and rollback;
3. preflight checks authority and execution readiness;
4. implementation feedback runs fast local checks;
5. independent validation evaluates the completed artifact;
6. pull-request validation binds CI and review to the current head SHA;
7. release validation checks deployment readiness and rollback;
8. production validation confirms health and the expected customer outcome.

Later evidence may invalidate an earlier conclusion. The factory must support correction without erasing history.

Review packages convert evidence into judgment

The operator should not reconstruct work from logs. A review package should show:

• original outcome and business reason;
• approved Plan and WorkOrder scope;
• files and systems changed;
• material decisions and deviations;
• criterion-by-criterion result with direct evidence links;
• verifier independence and artifact identity;
• failed, stale, waived, conflicting, or missing evidence;
• risk, uncertainty, reviewer focus, and rollback strategy;
• pull-request URL, branch, head SHA, CI, and merge state; and
• a recommendation with approve, reject, revise, and escalate actions.

The package summarizes. The underlying evidence remains available for audit and deep inspection.

8. Notes and lessons learned

My current conclusions are:

• Evidence is a product primitive, not a test log attached afterward.
• A criterion states a claim; a receipt records an observation; policy evaluates sufficiency; a human owns acceptance.
• Pass, stale, waived, unknown, and not applicable must remain distinct.
• Evidence belongs to an exact artifact and revision.
• Independence is established through systems and execution paths, not titles alone.
• Validator disagreement increases governance.
• Quality gates should fail closed while making remediation obvious.
• A review package should reduce human reconstruction work, not hide uncertainty.
• Production health and customer value are later evidence layers, not implied by merge.
• Strong quality systems create the conditions for greater autonomy.

9. Interview and discussion questions

1. Why is an artifact not automatically evidence?
2. What fields make a verification receipt trustworthy?
3. What is the difference between pass, stale, waived, and unknown?
4. Which changes should invalidate evidence?
5. How do you prove validator independence?
6. Why is a second model not sufficient by itself?
7. How should the factory handle conflicting validators?
8. When is evidence reuse safe?
9. What belongs in a review package?
10. Why should run completion not accept a WorkOrder?
11. How do waivers preserve flexibility without becoming bypasses?
12. How does quality engineering enable greater autonomy?
13. Which evidence should be required for a database migration?
14. What evidence proves validated customer value rather than deployment?