Sample article — starter content for NeuralSys.

Introduction

Every LLM application has a hidden layer nobody architected: the code that decides what text the model actually sees. I call it the context pipeline. When applications feel "flaky," the context pipeline is usually why.

The Problem With Prompts

A prompt is a snapshot. A context pipeline is a system. Snapshots don't handle:

  • Stale documents and contradictory sources
  • 200k-token windows filled with 190k tokens of noise
  • Tool outputs that arrive mid-reasoning
  • Multi-turn state that silently drifts

Designing the Pipeline

Think in stages, each with a contract:

Retrieve → Rerank → Deduplicate → Compress → Assemble → Budget

1. Retrieve broadly, then cut hard

Recall first, precision second. Pull 50 candidates, keep 5. Keyword + vector hybrid beats either alone for most codebases and docs.

def retrieve_context(query: str) -> list[Chunk]:
    dense = vector_store.search(query, k=30)
    sparse = bm25.search(query, k=30)
    merged = reciprocal_rank_fusion([dense, sparse])
    return reranker.top_n(query, merged, n=6)

2. Give every chunk a job

Each piece of context should answer one question: why is this here? Instruction, evidence, example, or state. If a chunk has no job, drop it.

3. Budget explicitly

const CONTEXT_BUDGET = {
  system: 1_500,
  retrieved: 6_000,
  history: 3_000,
  tools: 2_000,
  reserve: 1_500, // for the answer
};

When the budget overflows, truncate by priority — never by recency alone.

Measuring Context Quality

Log the assembled context for every Nth request and review it like code:

  1. Citation precision — does the answer actually use what you retrieved?
  2. Noise ratio — how many chunks were irrelevant?
  3. Freshness — how old is the newest evidence?
  4. Cost — tokens per grounded answer.

Key Takeaways

  • Prompts are UI; context pipelines are architecture.
  • Retrieve broad, rerank hard, budget explicitly.
  • Log and review assembled context the way you'd review SQL queries.