Evaluation Engineering, Trace Replay, and Run Comparison
Agent behavior changes when the model, prompt, tools, harness, context, environment, repository, or evaluator changes. A handful of successful demos cannot show whether a configuration is reliable, whether a new version is better, or whethe
A focused view of boundaries, contracts, state, authority, failure paths, and tradeoffs drawn from this chapter.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Design an evaluation program for an agent that repairs repository issues. Show task taxonomy, fixtures, dataset splits, baseline and candidate, repeated trials, deterministic and model graders, human calibration, trajectory capture, paired comparison, canary, rollback, and production feedback. Add one task with a flaky external dependency and explain how it is graded.
4. Tradeoffs and alternatives
Large evaluation suites improve coverage and increase cost, latency, and maintenance. Run fast deterministic gates on every change, representative agent cohorts on candidate changes, and expensive adversarial or long-running suites at risk-proportional intervals.
Golden answers simplify grading but can overconstrain valid solutions. Outcome and invariant-based grading allows implementation diversity but requires more careful fixtures and assertions. Live external dependencies improve realism and reduce repeatability; recorded dependencies improve comparison and may hide integration drift. Use both for different claims.
5. Current Mission Control Implementation
At study commit
d902fae,
Mission Control has context evaluations, deterministic learning signals,
dataset and experiment records, baseline/candidate comparison, independent
Verification Subjects and Plans, verifier Attempts, criterion-linked evidence,
exact-currentness checks, and Quality Gate Decisions. Run events, traces,
artifacts, model/token/cost fields, and inspector views provide material for
trajectory analysis.
The studied evidence does not establish a single end-to-end evaluation harness that reconstructs exact fixtures and environments, calibrates model graders, performs trace or simulation replay, computes paired statistical comparisons, and gates promotion across a representative production dataset. Production catalogs also lacked qualified execution routes, so repository mechanisms are not proof of an operating evaluation service.
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence