According to the no-free-lunch theorem, improvements in learning efficiency can only be achieved through the use of inductive biases.

Historically, applying classical PAC-Bayes theory to overparameterized models resulted in loose and unusable bounds because the compression term would dominate.

Deep neural networks exhibit "benign overfitting," meaning they can perfectly fit random noise yet still generalize well on structured data.

The volume of flat minima in a model's parameter space increases exponentially with the number of parameters, while the volume of sharp minima increases much less.

Research by Andrew Gordon Wilson shows that as the number of parameters in a model increases, it becomes possible to find more compressible solutions, which helps explain why overparameterization impr...

Research by Lotfi et al. found a negative correlation between the number of model parameters and the number of bits required to encode the training set.