Keep pulling the thread on Junyang Lin.
The training data for the Qwen series of models has scaled from 2 trillion tokens to 36 trillion tokens across its development versions.
The Qwen team has internally scaled their models to a 256,000 token context window during pre-training.
The Qwen 30B MoE model achieves performance roughly equivalent to their 14 billion parameter dense model.
The Qwen 30B MoE model has an inference speed comparable to a 7 billion parameter dense model.
The Qwen 3 Max model has a total parameter count of over 1 trillion.
The Qwen 3 Next model architecture uses a hybrid attention mechanism where for every four layers, three are a Gated Delta Network (a type of linear attention) and one is a full attention layer.
Using techniques like MInference and FP4 KVCache, the Qwen 3 Next model achieves a nearly 5x faster time-to-first-token (TTFT) and a nearly 3x faster time-per-output-token (TPOT) for decoding.
The Qwen team predicts that leading large language models will scale to 3 trillion, 5 trillion, and potentially over 10 trillion parameters in the future.
The Qwen 3 Next model is an 80 billion parameter MoE model that activates 3 billion parameters per token and was trained on over 10 trillion tokens.
The Qwen team adopted Mixture-of-Experts (MoE) models to balance performance and inference efficiency for their API on Alibaba Cloud, and to improve training efficiency for future model scaling.
Data scaling is more important than model scaling for improving large language models.
The Qwen team uses a multi-stage pre-training process where each subsequent stage uses a smaller but higher-quality dataset.