Keep pulling the thread on Y Combinator.
The speaker predicts that within one to three years, LLM inference will be viewed as a core capability that determines a model's peak intelligence, rather than just a cost or convenience factor.
Using the Speculative-Speculative Decoding (SSD) algorithm, it is possible to achieve a sampling speed of 300 tokens per second for Llama 3 70B on a system with four H100 GPUs.
Due to its factorized architecture, Diffusion Model Predictive Control (DMPC) can adapt to changes in an environment's dynamics by re-training only its dynamics model component on new data.
The speaker claims a venture associated with Yann LeCun raised $1.03 billion in March specifically to train world models.
The Latent World Model uses a SIGREG regularizer to prevent representational collapse by enforcing that latent space embeddings are Gaussian distributed.
The Latent World Model is approximately 50 times faster than competing world model architectures because its computations are performed in a lower-dimensional latent space.
Research by Andrew Gordon Wilson shows that as the number of parameters in a model increases, it becomes possible to find more compressible solutions, which helps explain why overparameterization improves generalization.
The amount of compute spent on pre-training large language models is growing by approximately 4x to 5x per year.
In data-constrained settings, it is more effective to train an ensemble of smaller models than to train a single large model with the same total parameter count.
A joint scaling recipe combining aggressive regularization and ensembling can achieve a 5x data efficiency win over standard pre-training methods.
Self-distillation, where a model is distilled into a new model of the same size, can significantly improve loss and outperform the asymptote of a heavily regularized single model.
In a continued pre-training scenario on math data, data-efficiency techniques like aggressive epoching and ensembling matched the performance of training on 73 billion tokens while using only 4 billion tokens, a 17x data efficiency gain.