An agent I built kept failing on the same class of task, and for two days I blamed the prompt. I rewrote it four times. I added examples. I moved the instructions to the end. Nothing moved the needle.
The actual problem was that by step nine the context window held the original request, eleven tool results, two full documents and a running summary — and the one line that mattered, a constraint the user gave in their first message, was buried somewhere around the 40% mark where the model was least likely to attend to it.
That is the shift the industry has been circling for about a year. Prompt engineering was about the words you write. Context engineering is about what occupies the window at the moment the model has to think — what you put in, what you leave out, what order it goes in, and what you throw away. On any system that runs more than a single turn, the second problem dominates.
The Window Is a Budget, Not a Bucket
The habit that causes the most trouble is treating a large context window as free storage. It is not free. It is a budget you spend, and every token you spend has three costs: money, latency, and the attention the model has left over for what actually matters.
That third cost is the one people miss. Models do not attend uniformly across their context. Information at the start and the end gets used reliably; the middle is where instructions go to die. This is well documented and it matches what I see in production — a constraint that works in a 2,000-token prompt gets ignored in a 60,000-token one, without any warning that it happened.
So the mental model I work with: every token in the window is competing for attention with every other token. Adding "just in case" context does not increase the chance of a correct answer. Past a point it decreases it.
Four Things That Fill Your Window
It helps to name the categories, because each needs different handling.
Instructions — the system prompt, the rules, the output schema. Stable across calls, cacheable, and worth keeping tight. This is where accumulated cruft hides; most system prompts I audit have three years of "and also don't forget to…" glued on.
Retrieved knowledge — documents, search results, database rows. The largest and most variable chunk, and the one most likely to be over-supplied. Ten retrieved chunks where three would do is not thoroughness, it is dilution.
History — the conversation or the agent's previous steps. Grows without bound if nothing manages it, and the oldest turns are usually the least relevant.
Tool results — API responses, file contents, command output. The sneakiest category, because a single unfiltered API response can be 8,000 tokens of JSON where the model needed four fields.
The Techniques That Actually Move Results
Compact tool output before it enters the window. This is the highest-leverage change available and almost nobody does it first. Your code called the API; your code knows the shape of the response. Extract what the model needs and drop the rest.
// Before: the whole payload goes in, ~6k tokens of mostly noise.
return JSON.stringify(await api.getOrders(customerId));
// After: the model gets what it can actually reason about.
const orders = await api.getOrders(customerId);
return orders.slice(0, 10).map((o) => ({
id: o.id,
date: o.created_at.slice(0, 10),
total: o.total_cents / 100,
status: o.fulfillment.state,
})); // ~300 tokens
On one project this single pattern cut average tokens per agent run by about 70% and improved accuracy at the same time, because the model stopped getting lost in fields it had no use for.
Structure the window in a fixed order. Instructions first. Retrieved material in the middle, clearly delimited and labelled with its source. The current task and the output schema last, immediately before generation. That final position is the most reliably attended part of the window and it should hold the thing you most need obeyed.
Summarise history rather than truncating it. Dropping the oldest turns loses the constraint the user gave in message one — which is exactly what bit me. A rolling summary plus the last few verbatim turns keeps the early decisions alive at a fraction of the token cost.
Keep durable facts outside the window entirely. If something must survive the whole session — the user's identity, their stated constraints, the current plan — put it in a structured state object that you re-inject in full on every call. Small, explicit, always present, and never at risk of being summarised away.
type AgentState = {
goal: string;
constraints: string[]; // "must not email the customer"
facts: { claim: string; source: string }[];
plan: { step: string; done: boolean }[];
};
Rendering that object into ~300 tokens at the top of every call is the difference between an agent that remembers the rules and one that quietly forgets them at step nine.
Retrieve less, but better. Five well-chosen chunks beat twenty mediocre ones. This is where a reranker pays for itself: retrieve broadly, then narrow hard before anything enters the window.
Order Your Prompt for the Cache
A practical detail with a direct effect on the bill. Providers cache long, stable prefixes — so if the first 4,000 tokens of your prompt are identical between calls, you pay a fraction for them.
This only works if the stable content genuinely comes first. Putting a timestamp or the user's name at the top of an otherwise fixed system prompt invalidates the cache on every single request. Stable content first, variable content last, always. It costs nothing to get right and I have seen it halve a bill on a chat product.
How to Debug This
When an AI feature misbehaves, the first question should not be "what's wrong with the prompt?" It should be "what exactly was in the window?"
So log it. The full assembled context, per call, with a token count per section — instructions, retrieved, history, tools. I keep this behind a flag in production and always on in staging. The number of times the answer has been immediately obvious from that log is the reason I now consider it non-negotiable.
The failures it surfaces are consistently unglamorous. A retrieval step returning the same chunk five times. A tool erroring and putting a 2,000-token stack trace into context. History quietly consuming 80% of the budget. A summary that dropped the one constraint that mattered. None of these look like prompt problems, and none of them are fixable by rewording anything.
Where This Leaves Prompting
Prompt engineering has not stopped mattering. Clear instructions, good examples, an explicit schema and a way for the model to say it does not know are all still worth the effort, and they are still the fastest wins on a single-turn task.
But the systems being built now are not single-turn. They loop, call tools, retrieve, and run for dozens of steps. In that setting the prompt is one component of the context, and usually not the one that is broken. The skill that separates an agent that works from one that almost works is knowing what to leave out.
My two days of prompt rewrites ended with a fix that changed no wording at all. I moved the user's constraints into a state object that got re-rendered at the top of every call. That was it.
