“Prior to the ST-MoE model, advancing state-of-the-art performance with sparse expert models was hindered by training instabilities and uncertain quality during fine-tuning.”