Progressive Delivery, Production Verification, and Rollback
Complete the factory path from release eligibility to observed customer outcome.
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
The chapter in one pass.
- Purpose: Complete the factory path from release eligibility to observed customer outcome.
- Best for: Release, reliability, product, platform, and engineering leaders.
- Prerequisites: CI/CD, Artifacts, Migrations, and API Compatibility.
- Reading time: 15 minutes.
- You will learn: How to design staged exposure, production verification, stop conditions, rollback, and outcome confirmation.
- Keep three ideas: deployment is a state transition, not success; rollback must be pre-engineered; and customer outcome closes the lifecycle.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Release a schema-dependent feature to ten percent of traffic. Add a delayed data-integrity defect, a healthy aggregate dashboard, and a failing high-value cohort. Show detection, containment, authority, rollback or forward correction, and evidence updates.
3. Enduring Principle
Model release states explicitly
eligible -> approved -> deploying -> deployed -> technically verified
-> activated -> outcome observing -> outcome confirmed
| |
v v
contained/rolled back/corrective work
Each transition binds artifact, configuration, environment, cohort, actor, policy, evidence, and timestamp. “Done” is not an authoritative state.
Use progressive exposure
Choose dark launch, internal cohort, percentage canary, region, tenant, feature flag, or blue-green strategy according to failure containment and representativeness. Predefine promotion gates, maximum exposure, observation duration, stop conditions, and decision owner.
Verify technical and intended outcomes
Technical verification includes availability, latency, errors, saturation, security, data integrity, dependencies, and synthetic journeys. Product verification checks the Mission’s expected behavior and guardrail measures. Segment results to detect harm hidden by aggregate averages.
Engineer rollback and containment
Rollback identifies reversible components, configuration, data, migration state, caches, consumers, and external effects. When reversal is unsafe, use containment and forward correction. Regular drills prove access, tooling, data restore, communication, and decision latency.
Feed production facts back into acceptance
Production evidence may supersede an earlier certificate, create a trust event, open corrective work, revoke a capability, or change the observation window. Preserve the original decision and append the new fact.
8. Notes and lessons learned
The factory’s strongest claim is not that it generated correct code. It is that it maintained an explainable, recoverable line from intent to an independently observed outcome.
9. Interview and discussion questions
- Which signals should stop a canary automatically?
- Why is technical health not customer value?
- When is rollback unsafe?
- How should delayed production failure affect prior evidence?
- What proves that a progressive-delivery strategy is representative?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence