arXiv Notify me• Feb 17, 2022paper

ST-MoE: Designing Stable and Transferable Sparse Expert Models

Noam Shazeer(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Noam Shazeer.

Efficient Scaling with Sparse Architectures Overcoming Training Instability

0 quotes

Concerns Raised

Managing training instability remains a critical design consideration for all sparse models.
Ensuring effective transferability and fine-tuning quality is a key hurdle for novel architectures.

Opportunities Identified

A clear architectural path to scale models to trillion+ parameters with manageable compute budgets.
Applying the ST-MoE design principles to other modalities like vision and multimodal AI.
Reducing the computational barrier to entry for developing and deploying state-of-the-art models.

Key Themes

Research Findings5

The Stable and Transferable Mixture-of-Experts (ST-MoE-32B) model contains 269 billion parameters.

The 269 billion parameter ST-MoE-32B model has a computational cost comparable to a 32 billion parameter dense encoder-decoder Transformer model.

The ST-MoE-32B model was the first sparse model to achieve state-of-the-art transfer learning performance on tasks including SuperGLUE, ARC Easy, ARC Challenge, XSum, CNN-DM, WebQA, Natural Questions, Winogrande, and ANLI R3.

Mixture-of-Experts (MoE) and Switch Transformers are an energy-efficient architectural approach for building larger and more capable language models.

Prior to the ST-MoE model, advancing state-of-the-art performance with sparse expert models was hindered by training instabilities and uncertain quality during fine-tuning.

Topics

Mixture-of-Experts (MoE)Sparse Models Switch Transformers ST-MoE Large Language Models (LLMs)Model Scaling Computational Efficiency AI Infrastructure Training Stability Transfer Learning Fine-tuning Natural Language Processing (NLP)SuperGLUE Question Answering Text Summarization

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

ST-MoE: Designing Stable and Transferable Sparse Expert Models

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Efficient Scaling with Sparse Architectures

Overcoming Training Instability

State-of-the-Art Transfer Learning

A Design Guide for Future Sparse Models

Research Findings5

Topics