Memory That Outlives Context

Design a memory service that lets an AI assistant remember a user across sessions that far exceed any single context window.

Requirements

Functional: capture each finished session as an episode; extract durable facts and episodic summaries from it; store them per user; deduplicate near-identical memories; detect and resolve contradictions when a newer statement supersedes an older one; decay or archive memories that stop being useful; retrieve the most relevant memories for the current turn; let a user list, edit, and hard-delete anything remembered about them.

Non-functional: strict per-user isolation (one user's memory must never surface in another's recall), p95 recall latency under 120 ms server side because this call sits in front of a model call, write path may lag the conversation by seconds, no memory permanently lost before extraction succeeds, and the store must survive reprocessing when the extraction model is upgraded.

Scale

2M daily active users, 6 sessions each per day (12M episodes/day). A raw session transcript averages 20 KB and caps at 256 KB. Extraction proposes about 4 candidate memories per episode and roughly 1.2 survive deduplication. Recall runs on about 30 turns per user per day (60M recalls/day) and asks for the top 20 memories out of a live set that stabilizes near 300 per user. Embeddings are 1024 dimensions. Assume 3x peak over average.

Out of scope

The assistant model itself, prompt assembly, tool calling, and billing.