According to the no-free-lunch theorem, improvements in learning efficiency can only be achieved through the use of inductive biases.

Historically, applying classical PAC-Bayes theory to overparameterized models resulted in loose and unusable bounds because the compression term would dominate.

Deep neural networks exhibit "benign overfitting," meaning they can perfectly fit random noise yet still generalize well on structured data.