Observability Semantics, Cost Attribution, and Forensics

3. Enduring Principle

Adopt shared semantic conventions

Define stable names and attributes for Mission, Plan, WorkOrder, Task, Attempt, Factory Version, capability versions, model route, tool call, sandbox, worker, repository, commit, candidate, evaluator, evidence, pull request, artifact, deployment, and outcome. Record status and failure class consistently across providers.

Correlate the complete lifecycle

Propagate trace and causation identity through API calls, queues, workers, tools, CI, artifact publication, deployment, and production observation. Human approvals and long pauses create linked spans or events rather than one misleading continuous span.

Control payload, privacy, and cardinality

Classify attributes before collection. Avoid secrets and unnecessary prompt or source content. Hash or tokenize identifiers where appropriate. Use bounded dimensions for metrics and retain high-cardinality detail in controlled traces or logs. Sampling must preserve errors, policy denials, security events, and representative successful runs.

Attribute usage and cost

Record input, output, cached, and provider-specific reasoning usage where available; model route; retries; tool and environment cost; verification; storage; and human attention. Roll these costs from Attempt to WorkOrder, workflow, tenant, and accepted outcome without double counting.

Create forensic bundles

For material incidents, freeze relevant manifests, identities, policy decisions, prompts or redacted hashes, context provenance, tool events, files, commands, network records, artifacts, evidence, approvals, and provider responses. Preserve integrity, access control, chain of custody, retention, and legal obligations.

Keep telemetry and evidence distinct

Telemetry supports diagnosis. A verifier converts selected observations into an evidence envelope tied to an exact subject, method, criterion, and producer. Raw trace presence does not satisfy acceptance automatically.

Stable semantic envelope

Every event or span records timestamp, event/span name, schema version, correlation and causation identifiers, tenant, environment, actor/workload, system/workflow/task/attempt identifiers, component and capability versions, state or operation, status, failure class, duration, usage/cost reference, and redaction state. Add subject digests for artifacts, contexts, evaluator results, evidence, and releases. Provider fields remain namespaced extensions.

OpenTelemetry generative-AI semantic conventions are developing. Pin an exact version, translate through an internal stable vocabulary, and test exporters before adopting changed attribute meaning.

Trace and event rules

Use spans for timed operations and links for asynchronous, fan-out, human-pause, and replay relationships. Use domain events for accepted facts and audit events for security-relevant actions. Propagate correlation through queues, providers, CI, delivery, and production; when a boundary cannot preserve trace headers, carry a signed or validated correlation field in the business contract. Clock uncertainty and missing segments are explicit.

Metrics, sampling, and alerting

Metrics use bounded labels such as workflow class, model profile, status, failure class, risk, and region. Store attempt, repository, user, and prompt identifiers in traces or events. Tail-sample failures, policy denials, high-risk actions, latency/cost outliers, and incident-linked runs; apply privacy and retention policy before export. Alerts name an owner, severity, baseline, threshold, window, deduplication key, runbook, suppression rule, and closure evidence.

Forensic bundle schema

A bundle includes manifest and exact versions; identities, grants, and policy decisions; ordered domain/audit events; selected trace/log segments; context and source lineage; tool calls and side-effect receipts; artifacts and digests; evaluator and proof records; human decisions; cost; incident controls; and known gaps. It is immutable, access-controlled, redacted by policy, and bound to a case. A bundle preserves observable facts, not hidden reasoning.

8. Notes and lessons learned

Observability becomes architecture when semantic choices determine whether runs, providers, workflows, and outcomes can be compared. Instrumentation without a shared subject model produces expensive anecdotes.

9. Interview and discussion questions

1. Which identifiers should propagate through every boundary?
2. How do you control high-cardinality dimensions?
3. What AI usage should be attributed to one accepted outcome?
4. When should a trace become evidence?
5. What belongs in a forensic bundle but not a routine log?