Language models learn to skip most context without losing much accuracy

Researchers have introduced a protocol called Declarative Attention that lets a model declare where it needs to attend, cutting the tokens read during decoding roughly in half with only a negligible hit to performance.

The problem: full key-value cache scans at every step

Large language models devote most of their attention to a tiny slice of the context, yet the global attention mechanism still scans the entire key-value cache to find the relevant tokens. When a user asks about a detail buried at the start of a million-token conversation, the model must read the whole cache for every token it generates in the reply. Existing approaches try to solve this with lightweight proxy scores that pre-identify important tokens, but even that external pass costs O(N) at each decoding step.

The internal approach: the model declares where to go

The new work, published as a preprint on arXiv, asks a simple question: does the model already know which parts of the context matter? The answer takes shape as Declarative Attention (DA), a protocol that splits generation into three modes the model declares inside its own chain of thought: `<global>` for full attention, `<focus>` for a specific region, and `<local>` for the most recent output only. The inference engine treats those declarations like tool calls and skips reading most of the KV cache.

Results: 30-52% fewer tokens read

In zero-shot tests on 15 long-context tasks using off-the-shelf models — Gemma-4-31B and Qwen-3.6-27B — DA reduced total tokens read during decoding by 52.0% and 31.1% respectively. Accuracy dropped just 1.27 and 2.75 percentage points, and the gap narrowed as the model grew larger. The figures come directly from the evaluation described in the abstract; no additional benchmarks or hardware details were published.

What this means in practice: a new axis for sparse attention

The implication is that sparse attention can work without retraining and without external ranking components — the model navigates itself. The authors note that DA opens a new optimization axis, and that training-based methods could improve results further in follow-up work. For now it remains a proof of concept on closed-weight models of medium size; larger models were not tested and no inference code was released.