arXiv Notify me• Oct 30, 2025paper

Kimi Linear: An Expressive, Efficient Attention Architecture

Yang Zhilin(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Yang Zhilin.

Overcoming Attention Bottlenecks Performance Without Compromise

11 quotes

Concerns Raised

Claims require independent, third-party validation on a diverse set of benchmarks.
The complexity of the hybrid architecture (KDA + MLA) may pose challenges for training and fine-tuning.
The '3B activated / 48B total' parameter count suggests a Mixture-of-Experts (MoE) model, which can have its own deployment and training complexities.

Opportunities Identified

Significant reduction in inference costs and latency for long-context applications.
Potential to become a new industry standard for attention mechanisms, replacing full attention in many use cases.
Enables new AI applications that were previously computationally prohibitive due to context length constraints.

Key Themes

Research Findings11

Kimi Linear is a hybrid linear attention architecture that outperforms full attention in short-context, long-context, and reinforcement learning (RL) scaling regimes.

A Kimi Linear model was pretrained with 3 billion activated parameters and 48 billion total parameters.

Using an identical training recipe, the Kimi Linear model outperforms a full Multi-Head Latent Attention (MLA) model by a sizeable margin across all evaluated tasks.

The Kimi Linear architecture reduces KV cache usage by up to 75% compared to full attention models.

The Kimi Linear architecture achieves up to 6 times the decoding throughput for a 1 million token context compared to full attention models.

Kimi Linear is designed to be a drop-in replacement for full attention architectures, offering superior performance and efficiency.

Kimi Linear's core component, Kimi Delta Attention (KDA), extends Gated DeltaNet with a finer-grained gating mechanism to more effectively use limited finite-state RNN memory.

Kimi Linear uses a chunkwise algorithm with a specialized Diagonal-Plus-Low-Rank (DPLR) transition matrix variant that reduces computation compared to the general DPLR formulation.

The pretrained Kimi Linear model is based on a layerwise hybrid of Kimi Delta Attention (KDA) and Multi-Head Latent Attention (MLA).

The Kimi Team has open-sourced the Kimi Delta Attention (KDA) kernel and its vLLM implementations.

The Kimi Team has released the pre-trained and instruction-tuned model checkpoints for its Kimi Linear model.

Topics

Kimi Linear Linear Attention Transformer Architecture Large Language Models (LLMs)Long-Context Models KV Cache Optimization Model Efficiency Inference Optimization Kimi Delta Attention (KDA)Gated DeltaNet vLLM Open Source AI Reinforcement Learning (RL)AI Hardware Efficiency Mixture of Experts (MoE)

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

Kimi Linear: An Expressive, Efficient Attention Architecture

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Overcoming Attention Bottlenecks

Performance Without Compromise

Architectural Innovation in Attention

Open Source and Practical Implementation

Research Findings11

Topics