Princeton researcher proposes architecture that closes decoder loop across successive tokens

Yifan Zhang of Princeton has published a technical report describing the Recurrent Looped Transformer, an architecture that passes the full decoder state from token to token — including the final output and the sliding-window attention (SWA) key-value cache in every layer. The report specifies an architectural blueprint, an implementation timeline, and a return contract for reinforcement learning, and it states explicitly that no efficiency metrics, reasoning quality measures, or scaling results have been measured. This is a design proposal, not a trained model with results.

RLT couples a causal encoder that runs in parallel with a recursive decoder that retains the full state Ht = (st, CtD), the final output and the KV caches of every decoder layer. For each token a gated merge combines the encoder representation et with the previous output s{t-1}, then each decoder block runs causal SWA over the decoder activations, cross-attention to the encoder memory, and an FFN. The window W includes the current token, so at most W-1 historical entries are retained per layer. The reference configuration uses 48 encoder layers and 48 decoder layers with shared attention and FFN weights, so each token executes 96 logical blocks, although the decoder blocks add cross-attention and therefore FLOPs per block are not equal. Zhang calls this parameter reuse, not activation copying.

Three design principles structure the proposal. The first offers unbounded temporal depth: after t tokens the state path from s0 has traversed t·LD decoder blocks (48t in the reference configuration) while per-token work stays constant. The report warns that gating and compression may suppress long paths, and that structural depth does not guarantee reasoning. The second principle addresses joint model-hardware design: encoder features and KV projections for known tokens use token-parallel kernels, decoder steps remain sequential within a sequence, but prepared updates from independent sequences can share a single stored kernel. The report states clearly that no exact parallel scan is assumed for the non-linear decoder, no prefill acceleration is claimed, and a standard parallel SWA decoder pass is not equivalent to recursion. Batching, kernel fusion, and checkpointing are listed as implementation targets, not as ready kernels. The third principle ties pre-training, instruction tuning, sampling, and reinforcement learning to a single state transition: the sampler logs the behavioural log-probability for every action under the actual sampling distribution, and the trainer reconstructs the encoder memory, the recursive output, and every SWA cache from the start of the sequence under the current parameters before scoring each action; previous rollout states are never reused. Claim 3.1 frames the trade-off: shifting the split between prompt and answer leaves the conditional distribution unchanged given a fixed token history.

Pretraining runs as next-token prediction on full sequences with full back-propagation through time. Instruction tuning masks the loss to assistant targets only but never masks state updates, so the assistant loss back-propagates through user and tool tokens. Appendix B shows that partial detachment is dangerous: the state-to-state Jacobian. The report discloses no experiments, benchmarks, or comparisons to existing models.