“Research using the FoMo-in-Flux benchmark investigated continual pretraining by analyzing data mixtures, stream orderings, various update methods including fine-tuning and model merging, meta learning rate schedules, and the effects of model and compute scaling.”