Keep pulling the thread on Yang Zhilin.
The Attention Residuals (AttnRes) architecture consistently outperforms the baseline Transformer architecture across all tested compute budgets.
The 'Block AttnRes' architecture matches the loss of a baseline Transformer model trained with 1.25x more compute.
A Kimi Linear model (48B / 3B activated, 1.4T tokens) with Attention Residuals (AttnRes) improved its score on the GPQA-Diamond multi-step reasoning benchmark by 7.5 points.
A Kimi Linear model (48B / 3B activated, 1.4T tokens) with Attention Residuals (AttnRes) improved its score on the HumanEval code generation benchmark by 3.1 points.
The 'Full AttnRes' architecture requires O(Ld) memory at scale.
The 'Block AttnRes' architecture partitions layers into blocks, uses standard residuals within each block, and applies attention only over block-level representations.
Using approximately 8 blocks, 'Block AttnRes' recovers most of the performance gains of 'Full AttnRes' and serves as a practical drop-in replacement with marginal overhead.
The Attention Residuals (AttnRes) architecture mitigates PreNorm dilution by keeping output magnitudes bounded across network depth.
The Attention Residuals (AttnRes) architecture causes gradient norms to be distributed more uniformly across layers during training.
Attention Residuals (AttnRes) is a drop-in replacement for standard residual connections in Transformers.
Attention Residuals (AttnRes) enables each layer to selectively aggregate earlier representations via learned, input-dependent attention over depth.