Model upgrade breaks agent memory, controlled study finds

An agent can retain the exact same memory pool and still forget everything when the underlying model is swapped. A new arXiv paper examines the phenomenon systematically: four memory-representation methods — long-context raw (LC-RAW), retrieval-augmented generation (RAG), compressed natural-language notes (NOTES), and fixed-schema knowledge graph (KG-fixed) — measured on 48 synthetic histories with random answer codes, exact-match scoring, and two open-weight models under 10 billion parameters. The central finding: only a fixed-schema structure survives a writer-model swap without losing accuracy.

Fixed-schema knowledge graph, the stable anchor

KG-fixed showed an accuracy shift of just +0.0004 ± 0.0020 after the writer model was replaced. In other words, the normalized, predefined structure does not depend on any single model's interpretation. It is the only one of the four substrates that enables true memory portability across versions without costly reconstruction or repair.

Compressed notes, tied to the model that created them

NOTES, by contrast, exhibited sharp dependence on the creating model. System accuracy moved asymmetrically: +9.91 or −13.28 percentage points depending on migration direction. Diagnostic decomposition showed that 80% (0.467 ± 0.014) of the deficit originated from information already lost at the initial construction stage; the compression itself discards context the new model cannot reconstruct. A repository-only fix failed to reach the 90% recovery target in any of the 48 test cases; preserving the raw history enabled recovery in 34 of 48 cases in the one direction tested.

RAG, partial embedding migration is not enough

In RAG systems, a mixed 50/50 index (half old embeddings, half new) yielded a 4.96-point improvement, while full re-embedding delivered 11.90 points. Retrieval failures accounted for 81% (0.364 ± 0.012) of the deficit — the old embeddings simply do not align with the new model's search space. The practical conclusion: isolating embedding spaces and a full re-run are necessary conditions, not recommendations.

What this means for production

The study underscores three operational principles: direction-specific migration testing (checking "upgrade" is insufficient; A→B and B→A must be validated separately), hard separation between embedding spaces of different versions, and retention of raw history as an insurance policy for memory repair. Without all three, a routine upgrade becomes a silent data-loss event.