Keep pulling the thread on Lilian Weng.
The Chinchilla scaling laws from Hoffmann et al. (2022) determined that for compute-optimal training, model size and the number of training tokens should be scaled in equal proportion, with optimal model size N_opt proportional to C^0.5.
The 70B parameter Chinchilla model, trained on 1.4T tokens, outperformed the 280B parameter Gopher model, which was trained on 300B tokens, under the same compute budget.
According to Pearce & Song (2024), the local power-law exponent for optimal model size scaling is close to Kaplan et al.'s 0.73 in the small-model regime but converges to Chinchilla's 0.5 as compute gets larger.
The total FLOPs (C) for training a Transformer model over D tokens can be approximated as C ≈ 6ND, where N is the number of non-embedding parameters.
Kaplan et al. (2020) found that larger models are more sample-efficient, reaching a given loss with fewer optimization steps and data points than smaller models.
The scaling laws from Kaplan et al. (2020) concluded that for optimal performance, model size should grow faster than dataset size, with the optimal model size N_opt proportional to C^0.73, where C is compute.
Kaplan et al. (2020) suggested that for a 10x increase in compute, model size should be scaled by approximately 5.5x and the number of training tokens by approximately 1.8x.
An analysis by Pearce & Song (2024) suggests the discrepancy between Kaplan et al. and Chinchilla scaling laws is partly due to Kaplan et al. excluding embedding parameters and experimenting on smaller models.
Hernandez et al. (2022) observed a double-descent phenomenon in test loss when training Transformer models on datasets containing a small fraction of repeated data.
Muennighoff et al. (2023) found that in data-constrained training, the value of excess model parameters decays faster than the value of repeated data tokens.
Lovelace et al. (2026) introduced an explicit overfitting penalty term to scaling laws for data-constrained training that grows with both the number of data repetitions and the model's capacity ratio (N/U_D).
Research by Lovelace et al. (2026) found that using strong weight decay reduces the overfitting penalty caused by data repetition in data-constrained training.