Context Engines: The System That Feeds the Model

Introduction

Every serious AI application eventually builds the same component: a service that takes a task and returns exactly the context the model needs — retrieved, ranked, deduplicated, budgeted, and cited. That component is the context engine. It sits between your data and every inference:

Data → Knowledge → Context → Reasoning → Action
                ↑
         context engine lives here

If agents are the new applications, context engines are the new databases.

What a Context Engine Does

Given a query and a budget, it returns an assembled context pack:

@dataclass
class ContextPack:
    chunks: list[Chunk]      # ranked evidence, each with source + score
    entities: list[Fact]     # structured facts from the knowledge graph
    history: list[Message]   # truncated conversation state
    tokens_used: int         # against an explicit budget
    citations: list[str]     # provenance for everything included

Five stages, each independently testable:

Stage by Stage

1. Retrieve from everywhere, in parallel

Vector search for semantics, BM25 for exact terms, graph lookup for entities in the query. Fan out, then fuse with reciprocal rank fusion.

2. Rerank and deduplicate

A cross-encoder reranker over the fused top-40, keep the top 6–8. Near-duplicate chunks (same source, overlapping text) collapse to one — duplicates waste budget and bias the model.

def build_pack(query: str, budget: Budget) -> ContextPack:
    candidates = fuse([
        vector_db.search(query, k=30),
        bm25.search(query, k=30),
        knowledge_graph.neighborhood(extract_entities(query)),
    ])
    ranked = reranker.top_n(query, dedupe(candidates), n=8)
    return assemble(ranked, query, budget)

3. Compress, don't just truncate

Long chunks get extractive summaries grounded in the chunk before assembly. A 2,000-token doc becomes a 200-token brief with a pointer to the full text.

4. Assemble by priority, spend the budget

const PACK_BUDGET = {
  entities: 800,    // structured facts first — densest signal
  evidence: 5000,   // reranked chunks
  history: 2500,    // conversation state
  reserve: 1500,    // room for the answer
};

Overflow rule: drop the lowest-priority chunk, never the citations.

5. Gate on policy

Before returning the pack: permission filter (can this user see each chunk?), freshness check (is anything expired?), and PII redaction. Policy at assembly time, not as an afterthought.

Operating It

A context engine is infrastructure, so it needs infrastructure habits:

Key Takeaways

• A context engine turns scattered sources into one budgeted, cited pack per inference.
• Retrieve broad → rerank hard → compress → assemble by priority → gate on policy.
• Log packs, measure noise ratio and citation precision, version everything.
• Build it once as a service; every agent and RAG app then shares it.