LIMe cuts transformer training compute by 15% without degrading model quality Other

Researchers introduced Layer-Integrated Memory (LIMe), a lightweight attention extension that trains large language models with 15% less compute while matching baseline performance.

In standard transformers, each layer relies exclusively on the hidden state of the immediately preceding layer, creating architectural pressure toward representation collapse. LIMe mitigates this bottleneck by routing cached key-value representations across earlier layers without expanding hidden-state size.

What it is

• Layer-Integrated Memory (LIMe). A modular mechanism that augments standard attention by learning head-specific and layer-specific routing weights to retrieve and mix key-value states from earlier layers.
• Mitigating representation collapse. By reusing past layer computations directly, LIMe preserves higher entropy across value vectors and enhances token separability throughout deep networks.
• 15% training FLOP reduction. In empirical evaluations on billion-parameter architectures, LIMe models reached the same training loss as standard LLaMA baselines while consuming 15% fewer compute operations.
• Benchmark performance. The approach outperformed competing architectures on synthetic arithmetic problems, multi-step reasoning tasks, and standard language modeling benchmarks.
• Open-source release. The authors published full model code, training scripts, and reproduction weights on GitHub under an open license following acceptance to EMNLP 2026.

Benchmark comparison

Sources

• arXiv: Layer-Integrated Memory for Transformer Architectures
• GitHub: corl-team/lime Repository