“The Chinchilla scaling laws from Hoffmann et al. (2022) determined that for compute-optimal training, model size and the number of training tokens should be scaled in equal proportion, with optimal model size N_opt proportional to C^0.5.”