“The trend in Mixture-of-Experts (MoE) models is towards increasing sparsity, with recent OpenAI models activating 4 out of 128 experts, compared to earlier Mistral models activating 2 out of 8.”