Quick Take
- The finding: Across 12 memory systems and two baselines on five workloads spanning 11 datasets, no single architecture dominates every scenario.
- Who wins where: Composite hybrids lead conversational QA; graph-based methods excel at single-hop factual recall but struggle with temporal reasoning.
- Updates break many systems: On knowledge-update and temporal-reasoning settings, exact-match and F1 scores vary from near-zero (some systems) to the mid-40s (stronger ones such as Zep).
Agent memory has grown from simple RAG into a full data-management stack: store, extract, retrieve, update and forget. Shanghai Jiao Tong University, Tsinghua and MemTensor researchers argue that end-to-end F1/BLEU scores treat that stack as a black box — so they open it.
What the study shows
They decompose memory into representation/storage, extraction, retrieval/routing and maintenance, then evaluate 12 systems (Mem0, Zep, Letta/MemGPT, A-MEM, LightMem, Cognee, MemTree and others) plus long-context and embedding-RAG baselines under RQs on effectiveness, retrieval fidelity, update robustness, long-horizon stability and cost.
- Backbone-robust when evidence is externalized: Effective systems stay robust across LLM backbones because they localize evidence before generation.
- Update table spread: Under knowledge-update settings, systems such as Letta score 0.0 on some exact-match columns while Zep reaches 44.4 on others — architecture choice dominates.
- Cost–performance frontier: Ablations of representation/storage show large latency spreads (sub-second to hundreds of seconds per query) alongside utility trade-offs.
- Guidance over slogans: The authors distill selection guidance: match topology to query type rather than chasing a universal memory product.
Why it matters: memory is infrastructure
Product teams pick a memory library and hope. This study shows the choice encodes trade-offs among factual recall, temporal reasoning, update safety and dollars per query.
Method in brief
Controlled module variants change one component at a time; end-to-end and fine-grained metrics; testbed to be released.
What remains open
- Moving ecosystem: Systems evolve quickly; rankings will shift.
- Table extraction noise: Some PDF tables are hard to parse exactly; cite primary tables for precise cells.
Context
Pairs with MemAdapter on sycophancy. See Agents.
What it means for agent builders
- Pick by workload: Graphs for facts; hybrids for conversational QA.
- Stress-test updates: Contradiction handling separates systems.
- Budget latency: Some designs are orders of magnitude slower.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “Are We Ready For An Agent-Native Memory System?” (arXiv:2606.24775v1, submitted June 23, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The paper is distributed under arXiv’s non-exclusive license, so i10X does not reproduce its figures; the chart was drawn by i10X from numbers reported in study design.
i10x Perspective
“Are we ready for an agent-native memory system?” The honest answer from this paper is: ready for specialized ones, not a one-size-fits-all brain. Treating memory as a database problem — modules, costs, update semantics — is overdue.
