Orchestration Failure, Recovery, and Cost Lab

1. Synthetic workflow

Build a three-step workflow: inspect a synthetic repository, generate a small change, and publish a mock review request. Pin workflow, model profile, tools, context, policy, evaluator, environment, attempt/tool/time/token/cost budgets, stop conditions, and approved fallback.

3. Failure injections

1. Duplicate one dispatch command and prove lease/state/idempotency controls prevent duplicate execution.
2. Return transient model errors long enough to open a circuit. Verify bounded jittered backoff and either approved fallback or explicit pause.
3. Make the publish tool time out after applying a mock side effect. Verify reconciliation before retry.
4. Feed repeated evaluator failures with no measurable improvement. Verify the loop stops and escalates instead of spending the remaining budget.
5. Exhaust the token or monetary reservation one unit before a model call. Verify no overrun and that safe teardown/containment capacity remains.
6. Crash the controller after a child result but before the join commit. On restart, reconcile the result exactly once.
7. Pause and resume after changing a governing context or capability version. Verify resume is denied and a new attempt or explicit replan is required.