Capability Evaluation, Certification, Promotion, and Retirement
Define how a capability earns, retains, loses, and exits production eligibility.
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
The chapter in one pass.
- Purpose: Define how a capability earns, retains, loses, and exits production eligibility.
- Best for: Quality, platform, AI, security, and governance leaders.
- Prerequisites: The first two Agent Factory chapters and Evaluation Engineering.
- Reading time: 13 minutes.
- You will learn: How certification differs from evaluation, how promotion stays risk-scoped, and how revocation propagates.
- Keep three ideas: certification is bounded evidence; promotion is a policy decision; and retirement is an engineered lifecycle.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Create a lifecycle for a code-modification skill. Promote it from one repository to an organization-wide low-risk lane. Then introduce a tool vulnerability and a model regression. Show separate containment, impact analysis, migration, and restoration decisions.
3. Enduring Principle
Use an explicit lifecycle
draft -> candidate -> evaluated -> certified -> canary -> generally eligible
| |
v v
restricted deprecated
| |
+------> quarantined -> revoked -> retired
Each transition identifies authority, scope, evidence, conditions, expiration, and rollback. Certification should name eligible task classes, risk levels, repositories or domains, environments, model and harness combinations, data classifications, and required human gates.
Evaluate four dimensions
- Functional: task success, correctness, structured outputs, recovery, determinism where required.
- Operational: latency, cost, rate limits, cancellation, observability, failure containment, resource cleanup.
- Security and policy: least privilege, prompt and tool abuse, data handling, provenance, dependency risk.
- Human factors: understandable plans, useful progress, actionable escalation, review burden, accessibility.
Evaluation results attach to exact package and environment digests. Aggregate scores must not hide hard-gate failures.
Promote progressively
Promotion expands eligibility through canary cohorts, observation windows, and explicit stop conditions. Compare against a frozen baseline using accepted outcomes, not only model-judge scores. A capability may be certified for read-only repository analysis and remain ineligible for code modification.
Retire without erasing history
Deprecation warns and provides migration. Quarantine stops use while facts are investigated. Revocation blocks new resolution immediately and may cancel or isolate active work according to risk. Retirement removes discoverability after migration while preserving historical resolution and evidence.
8. Notes and lessons learned
Certification is not a trophy attached to an agent. It is a temporary, scoped statement that a particular capability graph has sufficient evidence for particular use under particular controls.
9. Interview and discussion questions
- Why should certification expire?
- Which changes trigger immediate reevaluation?
- How does quarantine differ from revocation?
- What evidence justifies expanding a canary?
- How should active Attempts respond to a critical revocation?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence