arXiv Notify me• Jan 8, 2026paper

Excess Description Length of Learning Generalizable Predictors

Jan Leike(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Jan Leike.

Quantifying Learning with Information Theory Elicitation vs. Teaching in Language Models

0 quotes

Concerns Raised

The computational cost and practical scalability of calculating EDL for state-of-the-art large language models is not addressed.
The framework's applicability may be limited to scenarios where online, sequential training is feasible or a reasonable approximation.

Opportunities Identified

Applying EDL to evaluate the safety and predictability of fine-tuned models before deployment.
Using the framework to develop more efficient training and fine-tuning strategies that maximize generalization gain.
Formalizing the analysis of whether a model has truly learned a new skill or is simply pattern-matching on the training format.

Key Themes

Research Findings7

The Excess Description Length (EDL) metric provides bounds on a model's expected generalization gain.

The Excess Description Length (EDL) framework provides a rigorous foundation for the empirical observation that capability elicitation and teaching have qualitatively distinct scaling signatures.

A formal information-theoretic framework called Excess Description Length (EDL) has been developed to quantify the predictive structure that fine-tuning extracts from a training dataset and incorporates into a model's parameters.

The Excess Description Length (EDL) metric is defined via prequential coding and measures the gap between the bits needed to encode training labels sequentially with an online-trained model and the encoding cost using the final trained model.

The Excess Description Length (EDL) metric is non-negative in expectation.

In the infinite-data limit, the Excess Description Length (EDL) metric converges to surplus description length.

Experiments using toy models demonstrate that training on random labels results in an Excess Description Length (EDL) value near zero.

Topics

Excess Description Length (EDL)Information Theory Machine Learning Large Language Models (LLMs)Fine-tuning Model Evaluation AI Safety Generalization Prequential Coding Minimum Description Length (MDL)Capability Elicitation Latent Capabilities Predictive Modeling Theoretical AI Scaling Laws

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

Excess Description Length of Learning Generalizable Predictors

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Quantifying Learning with Information Theory

Elicitation vs. Teaching in Language Models

Connecting Information to Generalization

Clarifying Learning Dynamics

Research Findings7

Topics