Model Routing, Evaluations, and Capability Selection
No model is best for every factory operation. Strong models cost more and may be slower. Fast models may lack tool use, context, reliability, or risk approval. Provider outages and rate limits make a single route fragile. Choosing solely by
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Design routing for PLAN, EXECUTE, REVIEW, LOCAL, and LONG_RUNNING lanes. Add budget exhaustion, provider rate limit, a weak canary, high-risk work, and a kill switch. Explain every fallback that remains forbidden.
3. Enduring Principle
Route by required capability and governed evidence
Resolve the lowest-cost approved route that satisfies risk, complexity, capabilities, context, tool support, availability, latency, budget, and quality floor. Cost breaks ties only among eligible routes.
Keep model catalog, policy, and decision separate
The catalog records provider identity, version, tier, capabilities, availability, deprecation, risk approval, and cost estimate. Policy defines lane pools, minimum quality, fallback, budget, canary, and kill switch. A routing decision freezes inputs, alternatives considered, rejection reasons, selected model, source, and policy version for one run.
Evaluate the complete agent configuration
A model evaluation is not portable without its prompt, tools, context, temperature, runtime, and verifier. Use representative WorkOrder cohorts and measure criterion-level validation, retry-free completion, human acceptance, latency, cost, policy compliance, and failure severity.
Benchmark scores are useful priors, not production promotion evidence.
Use canaries and rollback
New routes begin with a small comparable cohort. Promotion requires minimum sample size, stable quality, no critical policy escapes, and human approval. Suspend on defined failure patterns and retain the prior policy for rollback.
Preserve validator independence
Routing implementer and validator through the same model, prompt family, and context can create correlated failure. Independence may require different providers, methods, tools, or deterministic verification depending on risk.
Fail closed when no eligible route exists
Fallback may relax cost or latency, but not required capability, risk approval, or policy. “No eligible model” is an actionable blocked state, not permission to use an arbitrary default.
8. Notes and lessons learned
Model capability does not grant autonomy. Routing selects an eligible component inside the operating system; policy, evidence, and human accountability still govern the result.
9. Interview and discussion questions
- Why should cost break ties only after quality eligibility?
- What must a routing decision retain?
- How do you evaluate a model route fairly?
- When is provider diversity a safety requirement?
- Why can same-model validation be correlated?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence