Agent memory: episodic, semantic, and what to keep

The first agent you build has no memory beyond the current conversation, and that works for about a week. Then users come back expecting continuity, and you start bolting on memory: a database table, a vector store, a summary of past sessions stuffed into the system prompt. By month three, the memory layer has more failure modes than the agent itself.

The two kinds worth distinguishing

Episodic memory — what happened, when, in this conversation thread — is what most teams build first because it’s obvious. Semantic memory — what the user prefers, what facts persist across sessions — is what makes an agent feel intelligent. They’re different storage problems, different retrieval problems, and conflating them produces an agent that remembers everything and uses none of it well.

What to actually keep

Aggressively forget. The temptation is to log everything in case it’s useful later; the consequence is a context window full of noise that crowds out the model’s reasoning. Summarize old episodes into salient facts. Promote facts to semantic memory only when they’ve been confirmed across multiple sessions — single-mention “facts” are mostly user mistakes the agent should not be encoding.

Agent memory is a curation problem, not a storage problem. The teams that get this right have a forgetting policy, not just a remembering policy.