Context Windows Are a Budget, Not a Bucket

Large context windows invited a habit: when in doubt, put it in. The whole policy manual, the last forty turns of conversation, every tool schema the agent might conceivably need, twenty retrieved passages because ten felt thin. The window is big enough, so the reasoning goes, and the model will find what matters. It is the reasoning of a bucket, and it fails on every axis a production system is measured on.

It fails on quality, because models do not attend evenly across long inputs and near-duplicate or contradictory passages act as distractors. It fails on cost, because input tokens are billed on every call and an agent loop resends its context every turn. It fails on latency, because prefill time grows with input length before the first output token appears. And it fails on governance, because a window assembled by accretion is one nobody can describe afterwards. The alternative is to treat the context window as a budget: a fixed amount of attention, allocated deliberately across named slots, each with a cap, a priority and a rule for what gets cut first.

Why the bucket fails#

The quality failure is the least intuitive. Adding a relevant passage helps; adding a passage that is merely similar can hurt, because it gives the model a plausible wrong thing to cite. Published long-context evaluations have repeatedly found that information placed in the middle of a long input is used less reliably than information near the start or the end. Instructions degrade the same way: a system policy at the top of a large window competes with everything beneath it.

The cost failure compounds. Suppose a hypothetical claims-review agent carries 120,000 input tokens and takes fifteen turns per case. That is 1.8 million input tokens per case before a single output token. Prompt caching reduces the price of the repeated prefix, but it does nothing for the quality or the governance problem, and anything that varies per turn sits outside the cache anyway.

The governance failure is the one regulated firms feel last and most expensively. If your decision record cannot say which passages were in the window, in what order, and why those and not others, it cannot explain the decision. Context assembly is part of the decision, and it deserves the same discipline as the model call.

Slots, caps and eviction rules#

A budget starts by naming what the window is for. Most production prompts decompose into six slots:

• System and policy. Role, rules, refusal conditions and output contract. Small, stable, never truncated.
• Task. The specific request and its structured parameters.
• Tool schemas. Definitions of the tools available on this turn, and only this turn.
• Retrieved evidence. Passages from documents and records that the answer should be grounded in.
• Conversation memory. Prior turns, raw or summarised.
• Output reserve. Tokens held back for the answer, including any reasoning the model produces before it.

Each slot gets a floor, a cap, a priority and an eviction rule. The floor guarantees a slot is never starved by a higher-priority neighbour; the cap stops any slot from crowding out the rest.

Those caps sum to about 49,000 tokens against a 128,000-token window. That is the point. The headroom is not waste; it is the margin that keeps quality, cost and latency stable when inputs grow. The policy belongs in configuration, versioned and reviewed like any other control:

# context-budget.yaml: claims-review agent, policy v7
window: 128000
output_reserve: 4000
slots:
  system:   { priority: 0, floor: 1500, cap: 3000,  on_overflow: fail }
  task:     { priority: 1, floor: 500,  cap: 2000,  on_overflow: fail }
  tools:    { priority: 2, floor: 0,    cap: 6000,  on_overflow: drop }
  evidence: { priority: 3, floor: 8000, cap: 24000, on_overflow: drop }
  memory:   { priority: 4, floor: 2000, cap: 10000, on_overflow: summarise }
retrieval:
  candidates: 40            # over-retrieve, then rerank
  keep_after_rerank: 12
  per_subquestion_quota: 6000
  dedup: sha256_normalised_text
  placement: best_first_then_restate_task
provenance:
  tag_format: "[E{n} {sha8}]"
  write_manifest_to_decision_record: true

The allocator

The allocator is deliberately boring. It reserves output tokens and all slot floors first, then walks slots in priority order, packing each slot's items by their keep score up to the smaller of its cap and what the budget still allows. Items that do not fit are dropped, queued for summarisation, or cause a hard failure, depending on the slot's rule.

type Slot = "system" | "task" | "tools" | "evidence" | "memory";
type Item = { slot: Slot; ref: string; sha256: string; tokens: number; keep: number };
type Rule = { priority: number; floor: number; cap: number; onOverflow: "fail" | "drop" | "summarise" };
type Policy = { window: number; outputReserve: number; slots: Record<Slot, Rule> };

export function allocate(policy: Policy, items: Item[]) {
  const order = (Object.keys(policy.slots) as Slot[])
    .sort((a, b) => policy.slots[a].priority - policy.slots[b].priority);
  const floors = order.reduce((n, s) => n + policy.slots[s].floor, 0);
  let flex = policy.window - policy.outputReserve - floors;
  if (flex < 0) throw new Error("slot floors exceed the window after output reserve");

  const packed: Item[] = [], toSummarise: Item[] = [], dropped: Item[] = [];
  const seen = new Set<string>();
  for (const slot of order) {
    const rule = policy.slots[slot];
    const limit = Math.min(rule.cap, rule.floor + flex);
    let used = 0;
    for (const item of items.filter((i) => i.slot === slot).sort((a, b) => b.keep - a.keep)) {
      if (seen.has(item.sha256)) { dropped.push(item); continue; } // content-hash dedup
      if (used + item.tokens <= limit) {
        packed.push(item); seen.add(item.sha256); used += item.tokens;
      } else if (rule.onOverflow === "fail") {
        throw new Error(`slot ${slot} needs more than ${limit} tokens; refusing to truncate`);
      } else {
        (rule.onOverflow === "summarise" ? toSummarise : dropped).push(item);
      }
    }
    flex -= Math.max(0, used - rule.floor);
  }
  return { packed, toSummarise, dropped };
}

The keep score is computed by whoever produces the items: reranker score for evidence, recency for memory, relevance to the current step for tool schemas. Two behaviours matter. The system slot fails loudly rather than truncating, because a silently clipped policy is worse than no response. And every eviction is returned, not discarded, so it can be logged against the decision.

Filling the slots#

A cap says how much a slot may hold. What goes into it, and in what order, is the other half of the budget.

Retrieval topology and dedup

How evidence reaches its slot matters as much as how big the slot is. Three patterns earn their keep.

Over-retrieve, rerank, then pack. Pull a wide candidate set cheaply, rerank it with a stronger model, and let the allocator take the best that fits. The cap, not the retriever's top-k, decides how much evidence the model sees.

Quota per sub-question. A compound question — is this procedure covered, and was prior authorisation obtained? — should retrieve per sub-question with a token quota each, so one well-documented sub-question cannot consume the whole evidence slot.

Dedup by content hash. The same paragraph arrives from a policy PDF, its HTML twin and a chunk that overlaps its neighbour. Hash normalised text (whitespace collapsed, Unicode normalised) and keep the first occurrence. Exact hashing is cheap and catches most duplicates; add shingle-based near-duplicate detection only when measurement says you need it.

Compression with provenance

When memory or evidence overflows, summarisation is tempting and dangerous. A summary is a derived artifact produced by a model, and it can drop the one qualifying clause that mattered. Two rules keep it honest.

Every summary carries the hashes of the items it replaced, so lineage can walk from the summary back to the source turns or passages. Cache summaries by those source hashes, so the same history compresses to the same summary rather than drifting between calls.

Prefer extraction over abstraction for evidence. Selecting the sentences that answer the question preserves wording that can be quoted and verified; paraphrasing a coverage exclusion invites the model to reason over a paraphrase. Never summarise the system and policy slot. If it does not fit, the policy is too long.

Anything the model could have relied on must be something you can name afterwards.

Position effects

Order is part of the budget. A pattern that works well in practice: policy first, then evidence with the highest-ranked passages nearest the top of the evidence block, then memory, then the task restated at the end so the question sits closest to where generation begins. Treat that as a starting hypothesis, not a law. Position sensitivity varies by model and by task, so placement belongs in the evaluation suite, and a model upgrade is a reason to re-test it.

Measure marginal utility with ablations#

Caps should come from measurement, not intuition. For each slot, run a fixed evaluation set at the full cap, at half, at a quarter and at zero, and plot task quality against tokens spent. For evidence, add leave-one-out ablations: remove each packed passage in turn and see whether the answer changes. Passages whose removal never changes an outcome are candidates for a lower rank or a tighter cap.

Suppose the evidence slot shows quality flattening beyond roughly 12,000 tokens for a given task. Everything above that point buys latency, cost and distractor risk for no measurable gain, so the cap moves down and the saved tokens either go unspent or go to a slot whose curve is still rising. Re-run the ablation whenever the model, the reranker or the corpus changes. Our quality practice treats these curves as monitored artifacts, not one-off experiments.

Provenance tags for every chunk#

The allocator's output is also governance evidence. Tag each packed item in the prompt with a short label and hash prefix, so the model can cite it and a reviewer can trace the citation. Then write the manifest — slot, source reference, content hash, position, token count, and every eviction with its reason — into the decision record. The window stops being an unknowable blob and becomes a list of named, hashed inputs, which is exactly what a decision ledger needs to explain what the model saw.

What to do on Monday#

• Measure what your production prompts actually contain today, broken down by slot, at the median and the 95th percentile.
• Write a versioned budget policy per workflow with floors, caps, priorities and eviction rules; reserve output tokens first.
• Make the system slot fail closed on overflow rather than truncating.
• Dedup evidence by normalised content hash, and give compound questions per-sub-question quotas.
• Summarise only with source hashes attached; prefer extraction for evidence.
• Run cap and leave-one-out ablations, set caps at the knee of the curve, and re-run on every model change.
• Tag every chunk and persist the context manifest, evictions included, with the decision record.