Runtime Orchestration and State Machines
Approving a WorkOrder does not execute it. Between authorization and a review ready result lies a distributed process that may run for minutes or hours, cross process and provider boundaries, survive restarts, wait for human decisions, and
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Draw dispatch through completion with Convex, Hono, an executor, a worktree, GitHub, and CI. Add a process crash after GitHub accepts a PR creation request but before Mission Control records the response. Show the idempotency key, provider reconciliation, and authoritative records that prevent a duplicate PR.
3. Enduring Principle
Orchestration coordinates authority; workers perform bounded execution
The control plane owns durable intent, policy, state, and decisions. The execution plane performs one authorized unit of work and reports structured facts. The orchestrator connects them without allowing an executor to approve its own work or invent its next authority.
Commands request; events report; state is derived under rules
A command expresses intent: dispatch, pause, resume, cancel, retry, approve. It may be rejected. An event reports an observed fact: process started, command completed, artifact produced, validation failed. Events do not automatically grant authority for the next action.
State machines define legal transitions and their guards. Useful runtime invariants include:
- no execution before an approved, current contract;
- at most one active Attempt for a Task unless parallelism is explicit;
- terminal records do not silently reopen;
- completion cannot bypass required validation or acceptance;
- cancellation prevents new work even if late success arrives;
- every external side effect has a stable idempotency key;
- retries create new history rather than rewriting failed history; and
- every transition identifies actor, reason, time, governing version, and evidence.
Persist before crossing an unreliable boundary
The safe pattern is to record the command or intent before dispatch, attach a stable identity, and reconcile the eventual result. This prevents an HTTP timeout from becoming uncertainty about whether work was authorized.
Exactly-once delivery is generally unavailable. Aim for at-least-once delivery with effectively-once effects: stable keys, atomic claims, conditional writes, and reconciliation against provider truth.
Model the lifecycle at several levels
Mission, WorkOrder, Task, Attempt, workflow step, tool call, pull request, and release have different state machines. A lower-level terminal state does not imply a higher-level decision. An Attempt may complete while its Task awaits review; a Task may finish while its WorkOrder lacks acceptance evidence.
Treat waiting as a real state
Paused, blocked, awaiting approval, and awaiting evidence are not failures. They identify what prevents progress and who or what can resolve it. Each wait state should contain a reason, owner, deadline, required action, and automatic resume behavior.
Reconciliation is part of orchestration
The control plane must periodically compare its records with executor, repository, CI, and delivery-provider facts. Reconciliation handles lost responses, late events, expired leases, stale approvals, and provider actions that happened outside the factory. It should repair projections or escalate ambiguity without erasing history.
8. Notes and lessons learned
The key distinction is between an agent loop and an engineering workflow. The loop reasons about the next action. The workflow owns durable progress and authority even when no model process is alive.
Mission Control already contains several sound mechanisms, but they do not yet compose into the completed production path. Architecture diagrams must not collapse “adapter exists” into “factory execution works end to end.”
9. Interview and discussion questions
- Why is exactly-once delivery usually the wrong promise?
- What must be committed before an executor starts?
- How do commands differ from events?
- What should happen when completion arrives after cancellation?
- Why must Task, Attempt, and WorkOrder state remain separate?
- When would you adopt a workflow engine instead of database-backed state?
- How do you prove restart safety?
- Which Mission Control orchestration claims are implemented, partial, and future?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence