arXiv Notify me• Apr 26, 2026paper

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Liang Wenfeng(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Liang Wenfeng.

Mixture-of-Experts (MoE) at Scale Innovations in Long-Context Efficiency

0 quotes

Concerns Raised

The practical hardware requirements for running inference on a 1.6T parameter model, even with sparse activation, remain substantial.
Potential for unforeseen failure modes or limitations in the novel architectural components (CSA, HCA, mHC) that are not covered in the preview.

Opportunities Identified

Enabling new applications that require processing and reasoning over entire books, large code repositories, or extensive financial reports.
Significant reduction in inference costs for long-context tasks, making them economically viable for a wider range of businesses.
Accelerating open-source AI development by providing a new, highly capable foundation model for fine-tuning and research.

Key Themes

Research Findings11

DeepSeek-V4-Pro is a Mixture-of-Experts (MoE) language model with 1.6 trillion total parameters and 49 billion activated parameters.

DeepSeek-V4-Flash is a Mixture-of-Experts (MoE) language model with 284 billion total parameters and 13 billion activated parameters.

The DeepSeek-V4 series of models, including DeepSeek-V4-Pro and DeepSeek-V4-Flash, support a context length of one million tokens.

The DeepSeek-V4 models were pre-trained on a dataset of more than 32 trillion tokens.

DeepSeek-V4-Pro-Max, a mode of DeepSeek-V4-Pro, outperforms its predecessors and sets a new state-of-the-art for open models in core tasks.

In a one-million-token context setting, DeepSeek-V4-Pro requires 27% of the single-token inference FLOPs compared to DeepSeek-V3.2.

In a one-million-token context setting, DeepSeek-V4-Pro requires 10% of the KV cache compared to DeepSeek-V3.2.

The DeepSeek-V4 series uses a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency.

The DeepSeek-V4 series architecture incorporates Manifold-Constrained Hyper-Connections (mHC) to enhance conventional residual connections.

The DeepSeek-V4 series uses the Muon optimizer to achieve faster convergence and greater training stability.

The model checkpoints for the DeepSeek-V4 series are available on the Hugging Face platform.

Topics

DeepSeek-V4 Mixture-of-Experts (MoE)Large Language Models (LLMs)Long-Context One Million Tokens Model Architecture Attention Mechanisms Compressed Sparse Attention (CSA)Heavily Compressed Attention (HCA)Manifold-Constrained Hyper-Connections (mHC)Muon Optimizer AI Training Inference Efficiency KV Cache Optimization Open Models

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Mixture-of-Experts (MoE) at Scale

Innovations in Long-Context Efficiency

Advanced Model Architecture and Optimization

Pushing the State-of-the-Art for Open Models

Research Findings11

Topics