Context engineering: a bigger window is not a strategy

Every few months a model ships with a larger context window and someone announces that retrieval is dead: just put everything in. This year the number is a million tokens. It is a genuinely useful capability, and the announcement is still wrong, for the same reasons it was wrong at 128k.

"Context engineering" is the name that has settled on the discipline of deciding what the model sees. I like the name because it makes the point: the context is something you engineer, not something you fill.

Why "put everything in" fails

• Attention is not retrieval. A model given four hundred pages and asked about one sentence will find it most of the time, and "most" is the problem. Precision degrades with distractors, and the failures are silent: a confident answer from the wrong paragraph.
• You pay for every token on every turn. A conversation that carries 300k tokens of "just in case" context costs the same whether the turn needed it or not. Caching helps; it does not make it free.
• Latency scales with input. Time-to-first-token on a huge prompt is a user-facing cost, and in an agent loop it is paid on every step.
• More context, more surface for injection. Every document you stuff in is text the model will read as if it mattered. Some of it was written by people who want the model to do something else.

The right use of a large window is headroom: room for the long trace, the big tool result, the document that genuinely needs to be read in full. Headroom you fill by default stops being headroom.

What belongs in the window

I find it useful to think of the context as four layers, each with a different lifetime.

Layer	Lifetime	Contents
Instructions	Stable across runs	Role, constraints, output format, the permission policy. Short, versioned, tested.
Task	This run	What the user asked, in their words, plus the structured facts you already know (IDs, dates, account state).
Evidence	This step	The minimal retrieved passages or tool results needed for the next decision. Cited. Swapped out as the task moves.
State	Accumulates	What has been decided, done, and learned so far. Compacted when it grows.

Most context problems are one layer leaking into another. Instructions that restate the task on every turn. Evidence that never gets swapped out, so step twelve is still carrying the documents from step two. State that is a raw transcript rather than a record of decisions. Naming the layers makes the leaks visible.

Retrieval is a ranking problem

If evidence is the layer that changes most, retrieval is the system that chooses it, and retrieval is a ranking problem before it is anything else. The question is not "do we have the document" but "is it in the top five for this query." Three things move that number more than anything else I have tried:

1. Chunking that follows the document's own structure. Sections, headings, table rows. Fixed-size chunks cut sentences in half and split the answer from the context that makes it an answer.
2. Hybrid search. Dense embeddings find meaning; keyword search finds the exact product code, error string, or name that embeddings blur. Running both and merging the results is cheap and it is almost always better than either alone.
3. Reranking. Retrieve fifty candidates with the fast method, then let a cross-encoder or a small model re-score them against the query and keep the top five. This step is where most of the precision comes from.

Then measure it. Take fifty real questions, mark which passages answer them, and compute how often the right passage is in the top k you actually send to the model. That one number — recall at k, on your own questions — tells you whether a bad answer is a retrieval problem or a generation problem, and it is the first thing I look at when quality drops. More on building that set in The evaluation gap.

Compaction: what to keep

Long-running agents accumulate state, and the raw transcript is the worst possible representation of it: verbose, repetitive, and full of dead ends that are no longer relevant but still influence the model. Compaction — replacing the past with a summary — is how you keep an agent running for hours without the context becoming the problem. APIs have started offering it as a primitive, which is welcome; whether you use the built-in version or your own, the hard part is deciding what survives.

My rule: preserve facts and decisions, drop deliberation.

• Keep: every identifier (order numbers, file paths, ticket IDs), every constraint the user stated, every decision made and the one-line reason, every action taken and its result, and anything the user corrected.
• Drop: the model's reasoning about options it did not take, retrieved passages that have already been used, intermediate tool output that has been summarized, and pleasantries.
• Never summarize away a correction. If the user said "no, the other account," that fact has to survive every compaction for the rest of the run. Corrections are the highest-value tokens in the transcript.

Compact on a trigger — a token threshold, a phase boundary, the end of a sub-task — rather than on every turn, and keep the structured state outside the transcript where you can. A JSON object of known facts is cheaper and more reliable than asking the model to remember them.

Citations as a contract

The last piece is the one that turns context engineering from a cost optimization into a trust mechanism. Every factual claim in the output should point at a piece of evidence that was in the window, and the user should be able to open it. That is not decoration; it is the contract that makes the system auditable. When a citation is missing, the claim is a guess, and it should look like one.

Designing for citations also disciplines the retrieval: if the model has to cite, you quickly find out which chunks are too small to cite meaningfully and which are so large the citation is useless. The two problems are the same problem seen from different ends.

Structure beats volume

A closing observation. In every system I have worked on, the biggest improvement in output quality came not from adding context but from formatting what was already there: clear section headers, consistent field names, tables instead of prose for tabular facts, and an explicit "here is what you know / here is what you need to decide" framing. Models read structure. Give them some.