Knowledge, Context, and Retrieval Pipeline Specification
Specify the governed path from registered source to immutable, attributable context package.
A focused view of boundaries, contracts, state, authority, failure paths, and tradeoffs drawn from this chapter.
1. Responsibility and boundary
The knowledge pipeline registers approved sources, ingests and transforms their content, maintains searchable representations, retrieves eligible candidates, and compiles the smallest sufficient context for one attempt. It does not grant tool authority, redefine business intent, or make untrusted source instructions governing.
The knowledge owner is accountable for source, connector, transformation, index, retrieval, and revocation contracts. Source owners retain authority for the underlying facts. Security owns access policy; workflow owners define task relevance; quality owns independent evaluation.
2. Pipeline and state
Source states are proposed, approved, active, degraded, suspended,
revoked, retiring, and deleted. Artifact states are processing,
indexed, stale, invalid, quarantined, and deleted. Checkpoints bind
connector version, source cursor, schema, transformation, and content digest.
Reprocessing is idempotent for the same source version and pipeline version.
4. Retrieval contract
- Resolve requester, workload, tenant, purpose, and data ceiling.
- Parse entities, exact identifiers, constraints, time, and required facts.
- Apply source eligibility, permission, tenant, lifecycle, and freshness filters before content reaches ranking or generation.
- Generate lexical, vector, graph, and metadata candidates under pinned index versions.
- Fuse and rerank using a versioned strategy appropriate to the task.
- Enforce diversity and group contradictions; do not discard governing counterevidence because it scores lower.
- Compile context by authority and token allocation, preserving citations and why each item was included or excluded.
- Freeze the package and bind it to the attempt.
Exact identifiers and code symbols often favor lexical search. Conceptual questions may favor embeddings. Relationship and blast-radius questions may need a graph. Hybrid retrieval is justified by measured improvement, not by default complexity.
7. Failure and recovery
| Failure | Detection | Runtime behavior | Recovery proof |
|---|---|---|---|
| Connector lag | Freshness SLO | Mark degraded; block freshness-critical tasks | Checkpoint catches up and gap scan passes |
| Schema change | Parser/contract error | Stop affected partition, preserve checkpoint | New parser version and reprocessing comparison |
| Permission mismatch | Negative authorization test | Deny candidate before ranking | ACL reconciliation and tenant isolation suite |
| Stale governing source | Authority/freshness rule | Exclude and block if required | Current source retrieved and package regenerated |
| Contradictory authorities | Contradiction group | Surface uncertainty and escalate | Named owner resolves or workflow records exception |
| Poisoning signal | Provenance, dominance, behavior anomaly | Suspend source and affected packages | Root cause, clean rebuild, red-team and regression tests |
| Index unavailable | Health/circuit state | Approved fallback or explicit unavailable state | Index restored and missed-change reconciliation |
10. Tradeoffs and nonclaims
Live source access maximizes freshness but increases dependency risk. Indexed copies improve latency and evaluation but add revocation and staleness work. Use live resolution for consequential facts and indexes for discovery where appropriate. This review-ready specification does not claim a production source registry, benchmarked ranker, deletion guarantee, or poisoning defense.
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence