Tasks, Attempts, Leases, Idempotency, and Recovery
Agent execution fails in ambiguous ways. A worker can disappear while its process continues. A retry can duplicate a commit or pull request. A stale worker can report success after another worker has taken over. If a Task and an execution t
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Draw two workers racing to claim one Attempt. Add lease expiry, a stale heartbeat, a GitHub PR created before a network timeout, and a replacement worker. Show the fence, provider reconciliation, immutable events, and the point where human escalation becomes necessary.
3. Enduring Principle
Preserve the Task; append Attempts
The Task retains objective, scope, dependencies, and lifecycle. Each Attempt freezes executor, model, context, policy, base SHA, worktree, budget, start time, and recovery relationship. Failed, timed-out, and canceled Attempts remain immutable evidence. Retry creates Attempt N+1.
A lease is temporary execution ownership
A worker atomically claims a pending Attempt with a lease owner, token or fencing generation, expiry, and heartbeat. Renewal extends ownership only when the caller still holds the current fence. Expiry makes the Attempt suspect, not automatically failed: reconciliation must determine whether external effects occurred.
A fencing token prevents a stale worker from writing after ownership transfers. Every material completion write should prove the current lease generation.
Idempotency identifies a logical operation
An idempotency key must be stable across transport retries and unique across
different logical operations. create-pr:{attemptId}:{headSha} is meaningful;
create-pr:{timestamp} defeats deduplication.
Idempotency belongs at every side-effect boundary: dispatch, event ingestion, commit, push, PR creation, approval, receipt, and webhook processing. The recorded result should be returned on replay. A database key cannot by itself deduplicate a provider call; reconciliation must use provider identity as well.
Recovery requires classification and a changed hypothesis
Retry is appropriate only when the failure is transient or a concrete input, environment, plan, or implementation has changed. Repeating the same action without new evidence wastes budget and can compound damage.
| Failure class | Default response |
|---|---|
| Authorization or policy | Stop and obtain valid authority; never retry blindly |
| Invalid configuration or context | Repair versioned configuration, then create a new Attempt |
| Capacity or rate limit | Back off within budget and deadline |
| Executor crash or lost lease | Reconcile external effects, then resume only if supported or create a new Attempt |
| Validation failure | Correct the defect through a new Attempt |
| Repository conflict | Rebase or replan with exact lineage |
| Unknown or contradictory evidence | Quarantine and escalate |
Cancellation is a protocol
Cancellation first prevents new work, then signals the executor, records acknowledgment or timeout, reconciles external effects, and terminates the Attempt. It cannot guarantee that already-issued provider calls vanish. Late events remain in history but cannot reopen authority.
Retry budgets are multidimensional
Bound attempts, time, cost, tokens, repeated failure signatures, and human interruptions. Exhaustion should create an attention item that explains what was tried, what changed, and why another automatic Attempt is unsafe.
8. Notes and lessons learned
The scheduler is an instructive example of progressive hardening. It proves Attempt identity and reasoned retry without pretending that state inspection is a lease. That precise language is more valuable than a broader autonomy claim.
9. Interview and discussion questions
- Why is a retry a new Attempt?
- What does a fencing token prevent?
- How do you reconcile a timeout after a provider side effect?
- Which failures must never be automatically retried?
- What makes an idempotency key stable?
- How does cancellation interact with late completion?
- What has Mission Control actually proven about Attempts?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence