Continuous Quality Contracts, Proof Packages, and Certificates

3. Enduring Principle

Build quality into every lifecycle transition

Use one continuous assurance loop:

Intent -> Plan -> Build -> Verify -> Validate -> Release -> Observe -> Learn

Each arrow is a governed transition. The current stage must produce sufficient evidence for the next stage; work does not advance merely because an agent or tool reports completion.

NIST's Secure Software Development Framework supports this lifecycle view by requiring security requirements, design review, code analysis, executable-code testing, provenance, vulnerability response, and continuous improvement rather than one terminal security scan.

Compile executable requirements into a Quality Contract

A Quality Contract is the machine-readable assurance specification for one version of a WorkOrder or release candidate. The agent does not decide what “done” means during execution. Human-owned intent, organizational policy, risk, and approved Factory Configuration compile the contract before mutation begins.

At minimum, the contract contains:

• subject identity: Mission, Plan revision, WorkOrder revision, repository, and intended environment;
• functional and non-functional requirements with stable identifiers;
• architecture, data, security, compliance, and dependency constraints;
• expected failure and recovery behavior;
• test and evaluation scenarios;
• required verifier capabilities and independence rules;
• evidence type, method, pass condition, freshness, and retention for every criterion;
• hard gates, scored dimensions, risk thresholds, and waiver rules;
• approval owners and separation-of-duty requirements;
• deployment, canary, rollback, and production-observation policy; and
• contract version, canonical digest, creator, approver, and validity window.

An abbreviated password-reset contract might look like this:

quality_contract:
  id: QC-WO-8241-R3
  subject:
    work_order: WO-8241
    revision: 3
    repository: example/accounts
    base_sha: 7b93c0d
    environment: production
  risk:
    class: HIGH
    reasons: [identity, credential-recovery, customer-data]
  requirements:
    - id: REQ-101
      claim: Reset tokens expire after 15 minutes and are single-use.
      required_evidence: [unit-test, integration-test, independent-security-test]
    - id: REQ-102
      claim: The API does not reveal whether an account exists.
      required_evidence: [negative-api-test, rate-limit-test, security-review]
    - id: NFR-201
      claim: Reset request latency is below 300 ms at p95.
      required_evidence: [performance-run]
  hard_gates:
    critical_security_findings: 0
    required_criteria_satisfied: 100%
    required_test_failures: 0
    rollback_strategy: required
    independent_security_receipt: required
  approvals: [engineering-lead, security]
  production:
    rollout: canary
    signals: [error-rate, latency-p95, email-failure-rate, abuse-rate]
    rollback_on: [slo-breach, security-event]
    observation_window: 7d

The contract is executable when policy can determine, without interpreting a completion narrative, whether every required proof exists and remains usable.

Model quality as Specification -> Evidence -> Decision

The central data structure is a requirement-to-evidence graph, not a folder of reports.

This resembles a structured assurance case: a claim is supported by an argument and attributable evidence, while counterevidence remains visible. The OMG Structured Assurance Case Metamodel provides a mature vocabulary for these relationships. The factory does not need to implement the complete standard, but it should preserve the distinction among claim, context, argument, evidence, and challenge.

For a tenant-isolation requirement, the graph might connect authorization code, unit tests, cross-tenant integration tests, a security review, the exact test run, deployment identity, and production access-control telemetry. The graph must also show what is missing, stale, waived, contradicted, or not applicable.

Treat generated code and generated evidence as untrusted

Builder output enters quarantine. Deterministic tools should establish the facts they can observe:

Compile -> Lint -> Static analysis -> Unit -> Contract -> Integration -> E2E -> Security -> Dependency -> Performance -> Build -> Provenance

Not every change requires every control. Policy selects them by affected surface, consequence, uncertainty, and environment. A documentation change and an authorization change should not consume the same assurance budget.

Tool output is still not self-authenticating. Each result needs the artifact digest, command or method, tool version, environment, inputs, producer identity, timestamps, status, and raw-artifact reference. SLSA provenance establishes where, when, and how a build artifact was produced. The in-toto Statement model binds a typed claim to immutable subjects by digest. These standards provide a sound envelope for factory evidence even though neither proves that the artifact is functionally correct or secure.

Separate verification from validation

Verification asks whether the artifact satisfies specified technical controls. It favors deterministic tools and reproducible execution.

Validation asks whether the delivered behavior solves the intended problem under realistic conditions. It may require domain judgment, adversarial scenarios, user research, probabilistic evaluation, or production comparison.

The producer cannot be the sole judge of either. Separate execution identity, environment, permissions, criteria, and receipts are more important than merely using a differently named agent.

Useful verifier capabilities include requirements coverage, testing, security, architecture, performance, accessibility, data migration, supply chain, and risk. They do not necessarily require eight permanently running agents. A factory may invoke deterministic tools, specialist agents, or qualified humans according to the contract. Adding agents without independent methods creates cost and correlated confidence, not assurance.

Test outcomes, failure, and recovery—not coverage alone

Coverage reveals which code executed; it does not prove correct assertions. The test portfolio should include the methods needed by the risk:

• unit, component, API contract, integration, end-to-end, and regression tests;
• negative, boundary, property-based, fuzz, and mutation tests;
• concurrency, retry, idempotency, timeout, cancellation, and recovery tests;
• performance, load, reliability, accessibility, and security tests; and
• migration, rollback, compatibility, and disaster-recovery exercises.

AI-enabled product behavior additionally needs versioned datasets and repeated trials for task success, hallucination, prompt injection, retrieval quality, tool selection, policy compliance, variance, latency, and cost. Agent evaluations should combine code-based, model-based, and human graders. Model graders require calibration against qualified humans; they are not independent truth authorities merely because they run in a separate process.

Anthropic's 2026 agent-evaluation guidance distinguishes tasks, trials, graders, assertions, and transcripts and recommends multiple evaluation layers. NIST AI RMF likewise calls for documented, repeatable test, evaluation, verification, and validation methods, production monitoring, and assessors who were not the front-line developers.

Separate Agent Trust, Artifact Trust, and Change Risk

These are different control inputs:

Agent Trust is historical evidence about a governed executor configuration: success, escaped defects, policy violations, retries, reviewer disagreement, rollback, evidence quality, and recovery.

Artifact Trust is confidence in one exact change and release candidate: criterion coverage, evidence strength, independence, freshness, provenance, reproducibility, and production behavior.

Change Risk is the consequence and likelihood of failure based on affected systems, data, reversibility, blast radius, uncertainty, and regulation.

Together they determine the maximum eligible autonomy. A trusted agent making a financial-calculation change still needs material human approval. A new agent may produce a low-risk artifact with strong evidence, but its limited history still constrains promotion.

Use scores for prioritization, hard gates for safety

A quality-confidence score can summarize dimensions for trend analysis and operator attention:

Q = f(requirements, tests, security, architecture, performance, observability, independent evaluation, provenance)

The formula, inputs, weights, uncertainty, and missing-data treatment must be versioned. Display a band with the dimension breakdown rather than false precision. Do not permit compensation across non-compensable controls.

Release eligibility is better expressed as a policy predicate:

eligible =
  all_required_criteria_satisfied
  AND all_hard_gates_pass
  AND no_blocking_counterevidence
  AND evidence_is_current_and_artifact_bound
  AND required_approvals_present
  AND confidence_meets_risk_threshold

A 98/100 score cannot override one critical security finding, missing authorization test, unknown migration result, or absent approval. GitHub's own artifact-attestation guidance makes the same conceptual distinction: provenance supports integrity and origin decisions but is not a guarantee that an artifact is secure.

Produce a Quality Proof Package and a bounded certificate

The Quality Proof Package is the complete reviewable assurance case for one release candidate. It contains:

• Quality Contract and digest;
• exact repository, source, diff, build, package, and deployment identities;
• requirement-to-evidence coverage graph;
• plan, architecture, security, migration, and rollback reviews;
• deterministic verification results and raw artifacts;
• independent validator receipts and disagreements;
• AI evaluation dataset versions, trials, graders, and uncertainty;
• risk classification, score breakdown, hard-gate results, waivers, and approvals;
• signed build provenance and dependency/SBOM references;
• release, canary, production, and customer-outcome evidence; and
• invalidation, supersession, incident, and corrective-work lineage.

The Software Quality Certificate is a concise, signed projection of that package. It identifies the subject by digest, contract version, claims met, blocking findings, assurance band, policy decision, approvers, issuer, issuance time, expiry, and proof-package digest.

The certificate means only:

At issuance time, this exact subject satisfied this version of the governed Quality Contract using the referenced evidence and approvals.

It does not mean defect-free, permanently safe, or valid for a different artifact or environment. New code, dependency changes, expired evidence, counterevidence, incident linkage, or policy revision can revoke or supersede it. A certificate without verified provenance and accessible evidence is decorative paperwork.

Continue the proof in production

Pre-release evidence cannot reproduce every workload, dependency, customer, or failure interaction. Release should progress through an explicit canary, observation, and expansion policy. Compare canary with a control using representative and attributable signals; also enforce absolute SLO limits.

Correlate traces, metrics, logs, deployment identity, feature configuration, and business outcomes so the factory can connect a production fact to the release and original requirement. OpenTelemetry supplies standardized signal categories and correlation mechanisms; it does not define the product's SLO or customer success measure.

When production contradicts pre-release evidence, the factory should contain or roll back the change, mark relevant evidence stale, reopen or create governed work, lower applicable autonomy, capture the missing scenario, and propose a regression test or evaluation update. Promotion of prompts, policies, datasets, or factory behavior remains human-governed.

8. Notes and lessons learned

My current quality doctrine is:

1. No assertion without evidence.
2. No evidence without provenance.
3. No acceptance without independent validation.
4. No autonomy without calibrated trust.
5. No release without a satisfied Quality Contract.
6. No score may override a hard gate.
7. No certificate means more than its exact subject, policy, evidence, and time.
8. No production contradiction may be hidden by an earlier pass.
9. No learning proposal may promote itself.

The factory produces software plus an assurance case. That is stronger than “CI is green” and more honest than promising defect-free software.

9. Interview and discussion questions

1. What can an AI Software Factory legitimately guarantee about quality?
2. How does a Quality Contract differ from acceptance criteria or a test plan?
3. What makes a requirement executable?
4. Why are verification and validation separate stages?
5. How do SLSA provenance and in-toto attestations support—but not prove—quality?
6. What belongs in a requirement-to-evidence graph?
7. How do you establish independent validation in a one-person company?
8. Why is code coverage insufficient?
9. How should probabilistic agent evaluations be represented?
10. What is the difference among Agent Trust, Artifact Trust, and Change Risk?
11. Why can a 98/100 release remain blocked?
12. What exactly does a Software Quality Certificate certify?
13. Which events revoke or supersede a certificate?
14. How should production evidence alter prior acceptance?
15. Which Mission Control components are canonical, partial, shadow, or mocked?