Data, Knowledge, Context, and Semantic Engineering

3. Enduring Principle

Preserve the handoff from source fact to model context

Every handoff should retain source identity, version or observation time, authority class, sensitivity, tenant, transformation lineage, and reason for selection. A model citation is useful only when the system can resolve it back to the exact source material that was used.

Engineer data fitness before retrieval

A source profile should answer:

• Is required data present, or is it missing?
• Is it current enough for this decision?
• Is its schema and meaning stable?
• Is it duplicated or contradictory?
• Which system and owner are authoritative?
• Which tenants, identities, and purposes may use it?
• Which transformations have been applied?
• How are correction, retention, and deletion propagated?

A Data Contract makes these expectations explicit: schema, semantics, quality thresholds, freshness, owner, sensitivity, lineage, allowed uses, and failure behavior. Missing data should produce an explicit unknown or blocked state when the workflow cannot proceed safely; it should not be converted into a confident default.

Build knowledge as a governed lifecycle

Knowledge Engineering includes source registration, connector identity, checkpointed ingestion, parsing, normalization, chunking, metadata enrichment, indexing, correction, reprocessing, and retirement. The system should know which source version and transformation produced every indexed unit.

Retrieval may combine:

• lexical search such as BM25 for exact names and uncommon tokens;
• dense vector search for conceptual similarity;
• metadata and relationship filters for scope and authority;
• hybrid fusion for complementary candidate sets;
• graph traversal for explicit relationships and lineage; and
• reranking to order candidates for the actual task.

No retrieval method is universally best. Code symbols, policy identifiers, natural-language concepts, and dependency relationships reward different methods. Evaluate the complete retrieval pipeline on representative queries.

Treat semantics as executable infrastructure

A controlled vocabulary defines preferred terms, aliases, and deprecated terms. A taxonomy organizes concepts. An ontology adds typed relationships and constraints. Entity resolution maps different source identifiers to one canonical entity while preserving source-specific identities.

A Semantic Contract should define canonical concepts, identifiers, allowed relationships, disambiguation rules, source mappings, owner, version, and compatibility policy. An unresolved term should remain ambiguous rather than be silently mapped. Semantic changes can invalidate retrieval results, context packages, evaluations, and downstream evidence.

Compile context for the decision, not the corpus

The context compiler begins with task requirements, policy, risk, actor, repository scope, model limits, and a token budget. It then selects, deduplicates, orders, compresses, and attributes material according to explicit rules.

Useful controls include:

• authority tiers separating approved contracts from reference material;
• recency and lifecycle filters;
• permission-aware retrieval before ranking;
• diversity controls that avoid ten near-identical chunks;
• contradiction detection and source comparison;
• token allocation by context class;
• compaction with retained decisions and unresolved issues;
• cache keys bound to source and policy versions; and
• “why retrieved” metadata for inspection and evaluation.

Context is an Attempt input, not an authority record. Retrieved text cannot alter the approved Mission, policy, tool grants, or acceptance criteria.

Evaluate each layer separately and together

Data evaluations measure completeness, freshness, validity, consistency, and permission correctness. Retrieval evaluations measure candidate recall, ranking, and source coverage. Semantic evaluations test aliases, identity resolution, ambiguity, and relationship correctness. Context evaluations test whether the final package is sufficient, minimal, current, attributable, and free of governing contradictions.

End-to-end task success remains necessary but is not diagnostic. A system that only records final success cannot determine whether improvement should target the source, ingestion, semantics, retrieval, context policy, model, or tool.

Make correction and revocation first-class paths

A corrected, deleted, reclassified, or compromised source must invalidate its derived artifacts, index entries, caches, context packages, and dependent evidence according to explicit policy. New work stops selecting affected material immediately. Running work is paused, cancelled, or allowed to finish only through a recorded risk decision. Historical packages remain reproducible under restricted retention but are marked ineligible for new decisions.

The system needs both forward lineage—what this source produced—and reverse lineage—which runs, decisions, and releases depended on it. Without both, revocation is an announcement rather than an enforceable control.

8. Notes and lessons learned

• Knowledge is prepared for reuse; context is compiled for one decision.
• A citation without source identity, version, and permission is decoration.
• The semantic layer should be as small as possible and as explicit as necessary.
• “The model missed it” is not a root cause until upstream layers are ruled out.

9. Interview and discussion questions

1. How do Data Understanding, Knowledge Engineering, Semantic Engineering, and Context Engineering differ?
2. When would lexical retrieval outperform vector retrieval for code work?
3. What must be retained to reproduce a Context Package?
4. How should the system behave when authoritative sources disagree?
5. Which semantic changes invalidate prior evaluation evidence?
6. Why must permission filtering occur before context reaches the model?