The Context Budget

Part I was about a machine you do not control. You cannot change how attention scales or how much memory a KV cache eats.

What you do control, completely, is what goes into the context. That is the subject of Part II, and this chapter is the framing for all of it.

The temptation, and why it fails

Context windows grew from 4,000 tokens to a million and more. The obvious conclusion — stop being clever, put everything in — is wrong, for three separate reasons.

It is quadratically expensive. Chapter 3’s table: 32x the context is roughly 1,000x the attention compute, and past about 100k tokens attention is the majority of what you are paying for. You feel it as latency and as an invoice.

It caps your concurrency. Chapter 5: context is per-user memory. A system where every request carries 200k tokens serves a fraction of the users of one that carries 8k.

And accuracy degrades. This is the one that surprises people.

Context rot

Models get worse as the context grows, even well inside the stated limit.

Part of this is attention: with more tokens, the weight any single token receives is spread thinner. Part of it is training — models saw far more short sequences than million-token ones, so the long end is less well-practised.

The related, older finding is lost in the middle: information placed in the middle of a long prompt is recalled less reliably than the same information at the start or end. Frontier models handle this much better than early ones did, but the gradient has not vanished. It has flattened.

The practical rule survives regardless of model generation:

The deeper reframe is this: a large window is a resource with diminishing returns, not free space. Your job is to find the smallest high-signal set of tokens that still lets the model act correctly.

Budgeting, concretely

Treat the window like any other fixed resource: allocate it in advance.

Component	Typical budget	Why
System prompt	500–1,000 tokens	Core logic and persona. Longer is rarely better.
Conversation history	2,000–5,000 tokens	The running state
Retrieved data	10k and up	The elastic part, sized to the task
Output reserve	1,000–4,000 tokens	Space for the answer and any reasoning

Five techniques for staying inside the budget

When a single request cannot be trimmed enough — and especially once agents enter in Part III, where context accumulates turn after turn — these are the five moves.

Compaction. When the history gets long, summarise it and restart the loop with the summary plus the few most recent artefacts. Keep the load-bearing details — decisions made, constraints discovered, bugs still open — and drop redundant tool output. Tune for recall first (keep everything that matters), then trim for precision.

Just-in-time loading. Keep lightweight references in context — file paths, URLs, row ids — and fetch the full content only when a step needs it. This is how a person works in a codebase: you hold the file tree in your head and open the one file you need, rather than reading the whole repository first. The cost is latency on each fetch, so hybrids are common: preload the obvious, fetch the rest.

Structured note-taking. The agent writes progress notes to a file or store outside the window, then reads them back when relevant. This is what lets a task run far longer than the window it runs in. Chapter 12 is about the storage side of this.

Sub-agent isolation. Hand a focused sub-task to a sub-agent with its own clean window; it returns a 1–2k token summary. The intermediate detail — twenty search results, a long file read — never touches the coordinator’s context. This is a context-management argument for multi-agent systems, entirely separate from any parallelism benefit.

System prompt calibration. Aim for the zone where the prompt is specific enough to be reliable but general enough not to be brittle. Use clear sections. Resist the urge to patch every failure by appending another sentence — that is how 500-token prompts become 6,000-token prompts that are worse.

Where the money actually is

Chapter 5 introduced prompt caching. It belongs here too, because it changes what “expensive” means.

The costly part of a long prompt is usually the stable part — a system prompt, tool definitions, few-shot examples — sent identically on every request. Cached, that content costs a fraction and skips most of prefill.

Which yields a layout rule that is worth more than most prompt tweaks:

[ stable ]  system prompt, tool defs      <- cacheable
[ stable ]  long reference docs that rarely change
[ varies ]  retrieved chunks for this query
[ varies ]  conversation history
[ varies ]  the user's message            <- never cached

Stable first, variable last. Anything variable near the top breaks the prefix match and forfeits the cache for everything after it.

Does retrieval still matter with a million tokens?

Yes, and it is worth being able to defend this crisply, because it is a standard interview question.

• Cost and latency. Re-reading a million tokens on every query is enormously more expensive than retrieving the five relevant chunks, even with caching.
• Freshness. Retrieval can hit a live API or a database updated a second ago. A context window holds a snapshot you pasted in.
• Scale. Enterprise corpora are terabytes. They do not fit in any window, and never will.

The honest nuance: for a genuinely small, stable corpus — under about 50k tokens — skipping the vector database and putting everything in a cached prompt is often the better engineering decision. Simpler, no index to maintain, no chunking decisions, no embedding model migrations.

Know which regime you are in. Building a retrieval pipeline for a 30-page handbook is over-engineering; putting a 4-terabyte document store in a prompt is not an option.

What to carry forward

Context is a budget with diminishing returns. Allocate it, reserve room for the output, put critical material at the edges, order it so the stable parts cache, and reach for the five techniques when it overflows.

The next three chapters are about the hardest line item in that budget: how you choose which few thousand tokens of a huge corpus deserve the space.