A user told our assistant, in their first message, that they were in Germany. Fourteen turns later it quoted them UK pricing.
Nothing had failed. The conversation history had grown past the trimming threshold, the oldest turns were dropped, and with them went the single most decision-relevant fact in the whole session. The model was reasoning perfectly over a context that no longer contained the thing it needed.
That is the memory problem, and it is not solved by a bigger context window. It is solved by deciding, deliberately, what a system remembers, where it keeps it, and how it gets back.
Models Have No Memory. You Are Building It.
Worth stating plainly because a lot of confusion starts here. A language model is stateless. Every call is the first call. The illusion of a conversation exists entirely because your code resends the history each time.
So "adding memory to an agent" is not a model feature you switch on. It is a retrieval and storage system that you design, and the design has four distinct layers that people tend to blur together.
The Four Layers
Working memory — what is in the context window right now. Bounded, expensive, and the only thing the model can actually see. Everything else exists to decide what goes in here.
Episodic memory — the record of this session. What was said, what tools ran, what came back. Grows fast, mostly irrelevant after a few turns, needs active management.
Semantic memory — durable facts about the user or the domain that should survive across sessions. "Based in Germany." "Prefers metric units." "Manages the Berlin team." Small, high value, retrieved by relevance.
Procedural memory — learned patterns of how to do things. Rare in production, mostly research territory, and I would not build it deliberately today.
The Germany failure was a category error: a semantic fact was living in episodic memory, so it aged out on the same schedule as small talk.
Pinned State: The Thing That Fixes Most Problems
The highest-return pattern, and the least sophisticated: keep a small, explicit, structured object of things that must never be forgotten, and re-render it in full on every single call.
type SessionState = {
goal: string;
constraints: string[]; // "based in Germany", "no email without approval"
entities: Record<string, string>; // orderId, accountId, ticketId...
decisions: string[]; // "chose the annual plan"
plan: { step: string; done: boolean }[];
};
// Rendered to ~250 tokens and placed at the top of every request.
function renderState(s: SessionState) {
return [
`GOAL: ${s.goal}`,
s.constraints.length ? `CONSTRAINTS:\n- ${s.constraints.join("\n- ")}` : "",
Object.keys(s.entities).length
? `KNOWN: ${Object.entries(s.entities).map(([k, v]) => `${k}=${v}`).join(", ")}`
: "",
s.decisions.length ? `DECIDED:\n- ${s.decisions.join("\n- ")}` : "",
].filter(Boolean).join("\n\n");
}
Two hundred and fifty tokens, present on every call, never summarised, never trimmed. That is what the Germany fix looked like — no new infrastructure, no vector database, just a decision that some facts are structural rather than conversational.
The other benefit is debuggability. When an agent does something odd, you can print the state object and see what it believed. A chat transcript makes you infer that.
Managing the Transcript
For the conversation itself, the pattern that has held up: keep the last few turns verbatim, and maintain a running summary of everything before them.
The detail that matters is when you summarise. Doing it on every turn is expensive and lossy — you are summarising a summary of a summary, and detail evaporates. Doing it only when you hit the limit means a large, disruptive compaction mid-conversation.
What works is summarising at a threshold, and summarising the original older turns rather than the previous summary, so each compaction is one lossy step rather than many stacked. Keep the summary in the same object as the pinned state, and re-summarise from source each time the window rolls.
One rule I have learned the hard way: never summarise the user's first message. It contains the actual request. Keep it verbatim, always.
Cross-Session Memory: Be Conservative
Long-term memory across sessions is where the interesting demos live and where most of the production pain is.
The naive version — embed every message, retrieve semantically similar ones later — fails in ways that are obvious in hindsight. It retrieves things that are topically similar but no longer true. It surfaces a preference the user changed months ago. It cannot distinguish "I use Postgres" from "I was thinking about using Postgres." And it leaks: a fact from a work session appearing in a personal one is unsettling even when technically correct.
What has worked better for me:
Extract facts, do not store transcripts. After a session, run a cheap model over it with a schema: what durable facts about this user did we learn? Store those as structured rows with a timestamp and a source, not as embedded chat blobs.
Make facts updatable and expirable. "Prefers email over Slack" should be replaceable, with the newest winning. Give facts a confidence and a last-confirmed date, and let stale ones decay.
Retrieve by category first, similarity second. If the current task is about billing, pull billing-related facts. Pure semantic similarity over a pile of memories returns plausible-looking noise.
Show the user what you remember, and let them delete it. Partly this is a privacy requirement in several jurisdictions. Mostly it is that a system with invisible memory feels creepy, and one with an editable list feels like a tool.
Scope memory to a tenant, enforced in the query. Not in the prompt. In the where clause. Cross-tenant memory leakage is a data breach, not a quality issue.
How to Tell It Is Broken
Memory failures are quiet — that is what makes them expensive. Two things that surface them:
Log the assembled context with a section token breakdown. Pinned state, summary, recent turns, retrieved memories, tool output. When something goes wrong you look at what the model actually saw rather than guessing.
Put multi-turn cases in your eval set. Most eval sets are single-turn, which tests none of this. A case that states a constraint at turn one and checks for it at turn fifteen is the only thing that would have caught the Germany bug automatically.
Start Smaller Than the Literature Suggests
If you are adding memory to an agent, the order I would go in: a pinned state object first, then transcript summarisation, then — only if there is a clear product reason — extracted cross-session facts. Skip the vector store over raw conversation history entirely.
Most memory problems I have debugged were not "we need a more sophisticated memory system." They were "the important thing was in a place that gets deleted." Two hundred and fifty tokens of pinned state fixes a surprising number of them.
