Continual Improvement Promotion Lab

Objective

Turn recurring production-like feedback into an evaluated, human-approved capability improvement without allowing the system to mutate active behavior directly.

Prerequisites and starting state

Prepare synthetic traces containing repeated human corrections, context misses, tool-selection errors, successful low-cost strategies, and one misleading outlier. Freeze the baseline capability graph and holdout dataset.

Required implementation

1. Normalize signals with source, subject, severity, attribution, evidence, and uncertainty.
2. Cluster recurring patterns while keeping the outlier separate.
3. Diagnose whether the smallest remedy belongs in deterministic code, prompt, skill, tool, context, route, evaluator, or workflow.
4. Create a versioned improvement candidate with hypothesis, risk, expected effect, experiment, guardrails, rollback, and owner.
5. Evaluate the candidate against development, regression, adversarial, and untouched holdout sets with repeated trials where needed.
6. Run a bounded canary or simulation and produce a promotion recommendation.
7. Require human approval before publishing a new capability version and preserve instant rollback.

Required failure

Include a candidate that improves the headline score while increasing unauthorized-action attempts or reviewer effort. The hard gate or guardrail must block promotion.

Evidence and pass criteria

Retain signals, clusters, diagnosis, candidate, frozen configurations, datasets, results, uncertainty, guardrail failure, approval, promoted version, and rollback proof. The lab fails if production feedback directly edits active instructions or the holdout set leaks into optimization.

Cleanup

Retire disposable candidate versions and preserve the experiment record.