Operational Autonomy and Trust Calibration

3. Enduring Principle

Operational law

Trust is earned through evidence, governed by policy, and continuously calibrated by outcomes—not by model capability.

A model upgrade does not automatically change authority. Sustained, observable, evidence-backed performance may justify promotion. Regressions, policy violations, weak evidence, or incidents must reduce the permitted autonomy.

Factory Operational Autonomy Levels

Level 5 does not remove humans. It moves them from individual routine decisions to policy, exceptions, and system governance. It should be rare until long-term evidence supports it.

Autonomy is scoped

Autonomy must be configurable per Factory, Mission, and WorkOrder. The effective level is the lowest applicable ceiling:

effective autonomy = min(factory, mission, work order, policy, trust)

The Factory ceiling describes the maximum authority of a configured factory. The Mission and WorkOrder ceilings reflect local intent and risk. Policy applies organizational constraints. The trust ceiling reflects current performance. No lower-level object may silently exceed its parent or policy ceiling.

The level is an upper bound, not an entitlement. A Level 4 factory may use one agent at Level 1 for a simple task. Greater capability does not require greater complexity or authority.

Promotion and demotion

Promotion requires explicit human approval based on sustained evidence. Demotion may occur automatically when policy or trust thresholds are breached. This asymmetry is deliberate. The system may fail safe without waiting for permission, but it may not grant itself more power.

A promotion policy must define a configurable evidence window. The initial Level 2-to-Level 3 standard requires at least 100 successful WorkOrders across at least 30 days of stable operation, at least 99 percent independent validation success, zero critical security or policy violations, zero unauthorized actions, and explicit human promotion. Both volume and elapsed time are required. Promotion must never rest on one successful run or a short streak. The retained decision must identify the evaluated scope, window, policy version, source records, and approver.

The exact demotion need not be one level at a time. A critical incident may reduce a scope from Level 4 to Level 1 immediately. Quarantine may reduce it to Level 0. Recovery requires corrected controls and a new record of sustained performance.

Factory Trust Score

The Factory Trust Score is a transparent control signal, not an opaque grade. The system should maintain an internal numeric score from 0 to 100 for calculation and trend analysis. Operators should normally see an interpretable trust band: Very Low, Low, Moderate, High, or Trusted. The numeric score is a measurement input; the band is the governance abstraction.

The initial trust bands are:

These thresholds must remain simple, stable, and versioned. An operator surface should explain the current band, its trend, the events that contributed to it, any hard override, and the evidence needed for review or future promotion. The score never overrides policy. It determines only the highest autonomy level the system is eligible to request.

Its components should include:

• production failures and rollbacks;
• security escapes;
• customer defects;
• validator disagreement;
• human overrides;
• policy violations;
• evaluated hallucination or unsupported-claim rate;
• recovery time; and
• evidence completeness, freshness, provenance, and quality.

The initial score uses five normalized dimensions:

Critical violations remain hard overrides rather than ordinary weighted deductions. This prevents a strong test history from averaging away evidence tampering, unauthorized action, or a serious security failure.

The initial score is calculated for a tuple of Factory Configuration version, repository, and workflow risk class. Executor and model are retained as diagnostic dimensions so a degraded component can be isolated without granting or removing authority at the wrong scope. A global average can hide an unsafe workflow behind many routine successes. Hard policy violations and critical incidents override the numerical score.

The score determines a trust ceiling. It never grants authority above policy, risk, Mission, or WorkOrder limits. Its inputs, weights, thresholds, source records, and changes must be inspectable and versioned.

Failure decay and retained accountability

Failures decay in scoring influence but never disappear from the audit record. The initial policy uses a rolling weighted window, with 90 days as the default. Recent behavior has greater influence than older behavior. Older failures lose weight gradually when sustained good performance follows, while critical failures remain permanent history. A repeated failure pattern resets the decay for that pattern. This permits earned recovery without rewriting history.

Trust-loss events

Immediate demotion or quarantine is justified by evidence that the system has lost trust, not merely by the existence of an ordinary execution failure. Qualifying events include:

• a security or policy violation;
• an unauthorized action or permission bypass;
• evidence tampering or required evidence that cannot be accounted for;
• hallucinated or fabricated results presented as fact;
• independent validation failure on a high-risk change;
• a customer-impacting production regression;
• repeated execution outside approved constraints; and
• unexpected behavior consistent with model drift or compromised tooling.

Permission bypass, evidence tampering, suspected compromise, and repeated constraint violations should quarantine the affected scope by default. Other qualifying events should demote it at least one level, with critical severity permitted to trigger quarantine. The system must retain the triggering event, previous and new ceilings, affected scope, policy decision, and required human review. A failed test, timeout, or bounded implementation error does not by itself prove loss of trust; severity, containment, truthfulness, and compliance with the authorized process matter.

Governance matrix

AI may prepare the decision packet and recommendation. It does not own these decisions:

The titles may vary by organization. The accountability cannot be assigned to an agent. Delegation of preparation or execution does not delegate the decision.

In a small organization, one person may combine Product Owner, Technical Lead, Mission Approver, Release Approver, and Factory Administrator. Implementation and independent validation must still remain logically separate through different execution contexts, evidence paths, or systems. Separation of duties should become stronger as organizational scale and risk increase.

Independent validation is technical, not merely organizational. Validators run through an execution path separate from implementation, do not reuse the implementer's claimed results, generate their own evidence against predefined acceptance criteria, and write immutable audit records. Human acceptance must review that independent evidence rather than trust the implementation agent's self-report.

The initial Level 2-to-Level 3 promotion requires approval from the Engineering Lead and designated Risk Owner. The Security Owner joins when the scope includes security-sensitive permissions or systems. In a small company, one person may hold more than one of these titles, but the decision must still rely on independently generated evidence and an immutable promotion record.

Validator disagreement increases governance

Validator results are evidence, not votes. Two passes and one failure do not produce a pass by majority. The factory must inspect the failed domain, method, severity, evidence, freshness, and independence. A security failure is not outvoted by two style checks.

Conflicting valid results create a Risk Review. The system preserves every receipt, identifies the conflict, blocks any action prohibited by policy, and presents the human owner with the evidence and safe options. It does not erase the disagreement through random retries.

The governing rule is simple:

Validator disagreement increases governance. It never decreases it.

Prevent approval theater

Humans should approve evidence and risk, not reconstruct routine execution from logs. A decision packet should show the governed requirement, acceptance criteria, test and quality results, security findings, performance change, regression status, coverage where relevant, risk, exceptions, recommendation, and uncertainty.

Routine evidence should remain inspectable without demanding deep attention. The factory escalates surprises: missing, stale, failed, contradictory, unusual, or high-risk evidence. High-risk work may still require code or architecture review. Evidence-focused approval does not eliminate professional judgment; it directs judgment to the parts that matter.

First proof workflow

The first workflow that should prove governed autonomy is Governed Issue -> Validated Pull Request:

1. A human creates a Mission.
2. An agent investigates the codebase.
3. The agent produces a versioned Plan.
4. A human approves that Plan version.
5. The factory executes the authorized WorkOrder.
6. Independent validators produce evidence.
7. Mission Control creates a review-ready pull request.
8. A human approves the merge.

This is first a Level 2 proof: the human defines and approves the work and reviews the material output. It approaches Level 3 only after sustained evidence shows that the factory can plan and execute reliably while humans intervene for material risk and accountable decisions. The proof deliberately ends before autonomous deployment. It exercises planning, authorization, execution, validation, evidence, governance, and oversight without expanding the initial safety boundary unnecessarily.

The first scenario adds a required Business Justification field to Mission creation. It touches the React UI, authoritative schema, validation, existing tests, browser testing, evidence generation, and pull-request lineage while remaining small enough to understand completely. See the Governed Issue to Validated Pull Request lab.

8. Notes and lessons learned

My current conclusions are:

• Autonomy belongs to a governed scope, not to a model.
• Model capability is evidence to evaluate, not authority to act.
• The effective level is always the lowest applicable ceiling.
• Promotion is a human decision. Demotion may be automatic.
• The factory computes a numeric Trust Score but communicates governance through explainable trust bands.
• Ordinary failure is not identical to loss of trust. Violating authority, truth, evidence, policy, or containment is categorically different.
• Promotion requires sustained scoped evidence, never one successful run.
• The initial Level 2-to-Level 3 bar is 100 successful WorkOrders across 30 stable days with at least 99 percent independent-validation success, no critical violations, no unauthorized actions, and human approval.
• Failure influence decays over a rolling weighted window, but audit history does not disappear.
• Validator disagreement is evidence of higher uncertainty and higher required governance.
• Humans should review the decision, evidence, risk, and surprise rather than reconstruct routine agent activity.
• A Trust Score must be transparent and subordinate to hard policy.
• Small organizations may combine accountable roles, but implementation and validation must remain logically separate.
• A factory that cannot reduce autonomy is not continuously governed.

Open questions:

1. Which event schema should represent trust changes, decay resets, and hard overrides?
2. How should confidence and sample sufficiency be shown beside the score?
3. Which controls prevent correlated implementation and validator failures?
4. How should a trust scope be reset after a material Factory Configuration change?

9. Interview and discussion questions

1. Why is autonomy an operational property rather than a model property?
2. Explain each Factory Operational Autonomy Level in one sentence.
3. Why should levels be scoped per Factory, Mission, and WorkOrder?
4. Why is the effective level the minimum of several ceilings?
5. Why may demotion be automatic while promotion remains human?
6. How would you keep a Trust Score explainable and resistant to gaming?
7. Why does majority voting fail for heterogeneous validators?
8. What should a Risk Review contain after validator disagreement?
9. How do evidence-focused approvals reduce approval fatigue?
10. Which decisions can an agent prepare but never own?
11. What evidence would justify moving a workload from Level 3 to Level 4?
12. Which Mission Control mechanisms exist today, and which remain future doctrine?
13. Why compute trust numerically while presenting operators with bands?
14. How does an ordinary execution failure differ from a trust-loss event?
15. Why is Governed Issue -> Validated Pull Request the correct first proof?
16. Why must failure influence decay while the audit record remains permanent?
17. Which controls make validation technically independent in a small company?