SemiAnalysis reveals what DeepSeek V4.1 Flash actually keeps in memory

Researchers at SemiAnalysis have peered inside the Engram gates of DeepSeek’s V4.1 Flash model and found that its internal memory stores far more than names and discrete facts. Engram — a mechanism that retrieves learned vectors for short token sequences — lets the model reuse familiar patterns instead of reconstructing them each time. The scan shows it learns to retain exactly what helps predict the next token, not what humans would consider worth remembering.

What Engram actually holds

The inspection surfaced names such as Ian Goodfellow, code snippets, task instructions and generic website boilerplate. Even copyright notices, licence text and sharing prompts appear in memory. The takeaway: the mechanism does not filter by human semantic importance. It filters by predictive utility. Dull, repetitive text that litters the web can serve as an efficient shortcut.

Different layers, different jobs

When the researchers looked at specific layers, a clear division of labour emerged. Layer 1 stores concrete phrases — “pints of frozen yogurt”, object descriptions. Layer 14, by contrast, holds reusable linguistic relations such as “three times as many”. The implication: lower layers specialise in entity recognition, while higher layers capture relational structures that can be applied across many contexts.

Implications for training-data cleanup

The fact that “boring” and legalistic text survives in Engram memory raises a question for data-cleaning pipelines. If generic, repetitive material supplies the computational shortcuts, aggressive removal of that material could degrade the mechanism’s efficiency — and affect the cost-benefit case for expanding Engram memory in future versions.

Built for serving with hardware in mind

Beyond what the memory stores, the architecture itself was designed with production inference in mind. Because retrieval is keyed by token IDs, it can be performed asynchronously and overlapped with other work, letting the system cut DRAM accesses without sacrificing throughput. In short: the memory is not only smart about what it keeps; it is also efficient about how it reaches the silicon.