DeepSeek (GitHub) Notify me• Jan 12, 2026code_release

DeepSeek ships Engram: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Liang Wenfeng(guest)

Get the full transcript next time DeepSeek (GitHub) releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Liang Wenfeng.

A New Axis of Sparsity: Conditional Memory The Economics of Model Scaling

7 quotes

Concerns Raised

The static nature of the Engram memory may limit adaptability to new information without retraining.
The optimal balance defined by the U-shaped scaling law might be task-dependent, requiring extensive tuning for different domains.
Integrating a novel, complex module like Engram into existing training and inference stacks could present significant engineering challenges.

Opportunities Identified

A new architectural paradigm for scaling LLMs by adding a dedicated, efficient knowledge lookup component.
Creation of highly knowledgeable models that are more efficient to serve due to the ability to offload static memory to host RAM.
The ability to explicitly trade-off computational capacity for memory capacity to optimize models for specific hardware and performance targets.

Key Themes

Research Findings7

A U-shaped scaling law governs the trade-off between neural computation (MoE) and static memory (Engram), guiding optimal capacity allocation.

The Engram-27B model demonstrates consistent improvements over Mixture-of-Experts (MoE) baselines across knowledge, reasoning, code, and math domains under strict iso-parameter and iso-FLOPs constraints.

The Transformer architecture lacks a native primitive for knowledge lookup, whereas Mixture-of-Experts (MoE) scales capacity via conditional computation.

The Engram module modernizes classic N-gram embeddings to achieve O(1) lookup time.

The Engram module relieves a model's early layers from the task of static pattern reconstruction, which may preserve the model's effective depth for complex reasoning.

The deterministic addressing used by the Engram module enables the offloading of its massive embedding tables to host memory with minimal inference overhead.

The Engram module augments a model's backbone by retrieving static N-gram memory and fusing it with the model's dynamic hidden states.

Topics

Large Language Models (LLMs)Model Architecture Sparsity Mixture-of-Experts (MoE)Conditional Computation Conditional Memory Engram N-gram Embeddings Knowledge Lookup Scaling Laws Inference Optimization System Efficiency Memory Offloading DeepSeek Transformer Architecture

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

DeepSeek ships Engram: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

A New Axis of Sparsity: Conditional Memory

The Economics of Model Scaling

Modernizing Classic NLP Concepts

System-Aware Model Design

Mechanistic Division of Labor

Research Findings7

Topics