Evaluation Engineering, Trace Replay, and Run Comparison

3. Enduring Principle

Evaluate the complete governed configuration

The evaluation subject is not a model name. It is a versioned combination of:

agent definition + model route + prompt + tools + skills + context policy + harness + environment + workflow + verifier

Changing any behaviorally relevant component creates a new candidate. The evaluation record should preserve the exact subject digest and distinguish it from the repository artifact being produced.

Use explicit evaluation records

An evaluation assertion should identify the claim, method, pass condition, required evidence, independence, and failure severity. Aggregate scores must not erase failed hard gates.

Build datasets from a task taxonomy

A dataset should represent the workflow's real distribution and important failure boundaries. Useful slices include task type, repository, language, risk, change size, environment, tool dependency, context size, ambiguity, failure mode, and required human intervention.

Include:

• normal cases that represent production frequency;
• boundary and adversarial cases that represent consequence;
• historical incidents and human corrections;
• negative cases in which the right result is to stop or escalate;
• recovery cases involving timeouts, unavailable tools, stale state, or partial effects; and
• held-out cases that were not used to tune the candidate.

Deduplicate semantically, not only by text hash. Track whether tasks or expected answers may have appeared in model training, prompts, examples, or prior optimization. Dataset growth should follow observed gaps rather than accumulate unreviewed production exhaust.

Combine graders without converting them into voters

Use deterministic graders for compilers, tests, schemas, paths, budgets, permissions, security scanners, artifact identity, and state invariants. Use model graders for bounded judgments such as plan completeness or explanation quality when deterministic methods are insufficient. Use humans for meaning, risk, unresolved disagreement, and grader calibration.

Calibrate a model grader against a human-reviewed set. Measure false positives, false negatives, disagreement by slice, sensitivity to presentation, and stability across repeated grading. Blind the grader to irrelevant candidate identity and self-justification. A grader produced by the same configuration as the candidate is not automatically independent.

Measure success, consistency, cost, and intervention together

Useful measures include:

• criterion and task success rate;
• retry-free success;
• failure and escalation correctness;
• policy compliance and unauthorized-attempt rate;
• human correction, override, and acceptance;
• latency, tokens, compute, and total cost per accepted outcome;
• severity-weighted regression rate;
• pass-at-k when several attempts are permitted; and
• consistency-oriented probability that all required attempts succeed.

Report confidence intervals and sample size. Prefer paired comparisons in which baseline and candidate run against the same task versions and comparable conditions. Segment before aggregating; a gain on easy documentation tasks must not hide a regression on high-risk migrations.

Distinguish inspection, replay, and re-execution

Trace inspection reads a retained event history without causing new effects. Recorded replay re-emits stored events into an inspector or test consumer. Simulation replay reruns control logic against recorded or mocked model and tool responses. Execution replay creates a new Attempt using a reconstructed manifest, fixture, environment, and external dependencies.

Execution replay is a new observation, not a rewriting of the original run. It may diverge because models, services, time, randomness, or network state differ. The system should record those differences rather than claim exact reproducibility.

Compare trajectories as well as outcomes

A run comparison should identify changes in context, prompts, tools, route, permissions, environment, tool-call sequence, retries, files touched, tests, latency, cost, policy decisions, human intervention, artifact, and evidence.

Trajectory diffs are diagnostic. They do not imply that one sequence is better merely because it is shorter. Tie findings to criteria: fewer tool calls may be efficient, or may indicate skipped investigation.

Connect offline, shadow, canary, and production evaluations

Offline evaluation is reproducible and safe but incomplete. Shadow evaluation uses production-shaped inputs without granting authoritative effects. A canary exposes a bounded cohort to the candidate. Production evaluation measures real outcomes, failures, interventions, cost, and drift.

Promotion should require a defined sample, quality floors, no critical policy regression, bounded uncertainty, rollback readiness, and human authority. Production incidents should create new test cases only after normalization, deduplication, privacy review, and expected-behavior approval.

8. Notes and lessons learned

• Reproducible inputs improve comparison; they do not make model output deterministic.
• A dataset is a governed product with owners, versions, privacy, and maintenance—not a folder of old prompts.
• Replay is most useful when its type and limitations are named explicitly.
• The most dangerous aggregate metric is one that hides the slice where consequence is highest.

9. Interview and discussion questions

1. What is the correct evaluation subject for an engineering agent?
2. How do a trial, grader, Evaluation Run, and Verification Run differ?
3. When is a model-based grader appropriate, and how would you calibrate it?
4. Why is execution replay a new Attempt rather than a replayed truth?
5. What should block promotion even when average success improves?
6. How do offline and production evaluations complement each other?