Memory for AI Agents: The Field Guide
Agent memory is any mechanism that persists information across sessions so experience converts into improvement: memory files, vector stores, state files, session summaries. The four working types are episodic (what happened), semantic (facts), procedural (how-to), and working memory (current task state). The design problem is not storage but staleness.
Key takeaways
- Memory is what makes agents compound instead of resetting to zero every session.
- Four types, four failure modes: episodic records intentions as outcomes; semantic goes stale; procedural drifts from reality; working memory leaks into long-term storage.
- Memory without decay handling is a liability: one bad write becomes an indefinitely repeated error.
- Recalled facts are claims to re-verify, not truths to trust. Date everything.
- Hook-written memory (audit logs) records what happened; agent-written memory records what the agent believes. Governance needs both.
Why memory is the compounding layer
Without memory, every session starts from zero: same questions, same mistakes, same context reconstruction. With it, experience converts into improvement, and the agent's hundredth session on your project is meaningfully better than its first. Memory is the layer that makes agents compound.
It is also the layer most likely to hurt you, because a memory system that stores errors repeats them indefinitely.
The four types
| Type | Stores | Typical form | Characteristic failure |
|---|---|---|---|
| Episodic | What happened | Session summaries, logs | Intentions recorded as outcomes |
| Semantic | Facts | Fact files, indexes | Staleness: yesterday's truth trusted today |
| Procedural | How to do things | Skills, runbooks | Drift from how things are actually done |
| Working | Current task state | Scratchpads, checkpoints | Leaking into long-term storage |
The procedural row explains a deep connection: skills are procedural memory made durable. The skill format is what procedural memory looks like when you take it seriously.
Storage and retrieval, honestly
The fashionable answer is a vector store; the correct answer at most project scales is flat files. Indexed fact files (one fact per file, a one-line-per-fact index loaded each session) are human-inspectable, git-versionable, cheap, and debuggable with grep. Semantic retrieval earns its complexity only at volumes plain files cannot serve.
Whatever the store, the retrieval design question is the same: what loads always (and taxes every session), what loads on relevance, and what waits for an explicit query. Index-always, bodies-on-demand is the pattern that keeps the tax flat as memory grows.
Staleness is the real design problem
Every memory system's failure modes reduce to one: information quality over time.
- Staleness. The port changed; the memory didn't. Defense: date every fact, and treat recall as re-verification. A recalled fact is a claim, not a truth.
- Poisoning. One bad write (wrong conclusion, or injected content) recalls as trusted fact forever. Defense: write rules about sources, editable stores, consolidation passes that prune.
- Unbounded growth. The index itself becomes the context tax. Defense: consolidation on a schedule; deletion as a first-class operation.
- Leakage. Secrets and personal data written casually, recalled promiscuously. Defense: privacy is a write-time property; redaction at read time is already too late.
The Memory Safety Model turns these into six assessable axes.
One structural insight
Agent-written memory records what the agent believes. Hook-written memory (an audit log appended by infrastructure the agent cannot edit) records what happened. The gap between them is exactly where false history lives: summaries that say "fixed" about things that were attempted. Governance-grade systems keep both and use the second to audit the first.
Start with indexed fact files, date everything, and schedule a consolidation pass before you need one.
Frequently asked questions
How does AI agent memory work?
Information persists across sessions in files, databases, or vector stores, then loads at session start or on retrieval. The four types are episodic, semantic, procedural, and working memory.
What is the biggest agent memory failure?
Staleness: facts recalled long after they stopped being true. Systems that date facts and re-verify on recall handle it; systems that load undated facts verbatim repeat one bad write forever.
Are vector stores necessary for agent memory?
No. At most project scales, flat files with an index outperform vector stores on inspectability, versioning, and cost. Semantic retrieval earns its complexity only at volumes plain files can't serve.
Summary
- Topic
- Memory for AI agents
- Definition
- Cross-session persistence converting experience into improvement, in four types: episodic, semantic, procedural, working
- Best used for
- Stopping repeated mistakes and re-derived context
- Related concepts
- memory safety, semantic memory, indexed fact files, audit log
- Common mistakes
- No decay handling, intentions recorded as outcomes, secrets written to memory, unbounded index growth
- Recommendation
- Start with indexed fact files, date every fact, treat recall as re-verification, add consolidation passes