Resilience, Disaster Recovery, and Factory SRE
Keep the factory safe and explainable when its own infrastructure fails.
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
The chapter in one pass.
- Purpose: Keep the factory safe and explainable when its own infrastructure fails.
- Best for: Reliability, platform, security, architecture, and executive owners.
- Prerequisites: Release, Production Feedback, and Factory SRE.
- Reading time: 15 minutes.
- You will learn: How to define failure domains, graceful degradation, disaster recovery, emergency authority, and resilience exercises.
- Keep three ideas: durable intent must survive worker loss; failover must preserve authority; and recovery is unproven until exercised.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Fail the primary control-plane region during active code publication. Add a delayed provider response and a revoked tool. Show fencing, recovery mode, state restore, external reconciliation, operator decisions, and proof that no duplicate publication occurred.
3. Enduring Principle
Define failure domains and criticality
Inventory components, dependencies, authoritative data, derived data, credentials, external effects, and recovery owners. Classify which capabilities fail closed, degrade read-only, queue safely, switch provider, or require emergency shutdown.
Set RTO and RPO by capability
Recovery time and recovery point objectives apply to control-plane state, evidence, registry, queues, artifacts, and telemetry separately. An RPO of zero for authority records may require synchronous durability; telemetry can often tolerate bounded loss.
Preserve authority across failover
Failover must not create two active authorities, reset budgets, reuse single-use permits, revive revoked capabilities, or duplicate publication. Use fencing tokens, generations, leases, idempotency, reconciliation, and explicit recovery mode.
Engineer graceful degradation
Examples include read-only operator access, stopping new admission while allowing safe cancellation, retaining events locally, switching to prequalified model routes, or delaying publication. Degradation is declared policy, not improvised fallback.
Exercise recovery
Run game days for provider outage, queue corruption, lost worker, expired credential, evidence-store failure, regional loss, and compromised capability. Measure detection, decision, containment, restore, reconciliation, communication, and return to service.
Control emergency authority
Break-glass access is narrowly scoped, time-limited, strongly authenticated, independently logged, and reviewed afterward. Emergency action cannot silently erase history or become the routine operating path.
Recovery contract matrix
| Subject | Authoritative or derived | Example RPO | Recovery behavior | Verification |
|---|---|---|---|---|
| Intent, policy, grants, decisions | Authoritative | Zero or near-zero by risk | Fail closed; restore ordered durable state | Integrity, sequence, signer, and policy checks |
| Workflow and attempt state | Authoritative | Bounded by checkpoint | Reconcile leases, commands, and external effects | State-machine invariant and orphan scan |
| Capability and system inventory | Authoritative | Small bounded loss | Restrict selection until current | Registry digest and dependency resolution |
| Artifacts and evidence | Immutable authoritative records | Zero after publication | Restore by digest with provenance | Hash, signature, retention, and subject binding |
| Search indexes and projections | Derived | Rebuildable | Rebuild from authoritative events/sources | Count, digest, and query comparison |
| Telemetry | Operational, partly lossy | Bounded | Restore collection; mark observation gap | Coverage and clock/correlation checks |
RTO and RPO are scoped to capability, region, risk, and failure mode. A single platform-wide number hides the records that cannot tolerate loss.
Replay, resume, and reconciliation
Replay reprocesses retained inputs in a controlled environment; it must not repeat external effects unless explicitly simulated or protected by an idempotency contract. Resume continues a paused workflow only after manifest, policy, grants, capabilities, context, leases, and budgets are revalidated. Reconciliation compares desired durable state with workers, queues, providers, artifacts, and downstream systems to classify each effect as absent, completed, failed, or unknown. Unknown is a first-class state requiring investigation.
Disaster and dependency scenarios
Exercise control-store corruption, regional loss, identity issuer outage, model/tool provider degradation, artifact/evidence unavailability, queue duplication, lost worker, compromised credential, and unavailable human approver. Failover uses preapproved identities and versions; it cannot bypass authority or evidence because the primary region is unavailable. Preserve forensic state before destructive repair where safety permits.
Factory SLO and error-budget policy
Define SLIs for admission, dispatch, state durability, control enforcement, successful reconciliation, proof-package availability, recovery time, and accepted-outcome reliability. Error-budget exhaustion reduces change and autonomy, prioritizes reliability work, and may restrict high-risk workflows. Safety and security incidents are not offset by good average availability.
8. Notes and lessons learned
Factory reliability is a safety control. When the system cannot prove current authority, configuration, or evidence, availability should yield to containment.
9. Interview and discussion questions
- Which factory components require an RPO of zero?
- How do fencing tokens prevent split-brain execution?
- What can remain available when admission is stopped?
- Which fallback changes require new approval?
- What evidence proves disaster recovery works?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence