Keep pulling the thread on Zhilin Yang.
A proper implementation of the MEUM optimizer can yield a 2x improvement in token efficiency.
Moonshot AI published the first work demonstrating that the MEUM optimizer is scalable for Large Language Model (LLM) training.
Moonshot AI successfully trained a 1 trillion parameter model, K2, using the MEUM optimizer with the QK-CLIP technique, marking the first successful large-scale training with this optimizer.
The KIMI-Linear architecture is the first of its kind to outperform full attention on short context tasks, long input tasks, and long output tasks.
The K2.5-based model was trained on over 15 trillion tokens with a stable training process that exhibited no loss spikes.
Kimi K2.5 is the first open model with native, jointly trained vision and text capabilities from the start of pre-training.
Using a "zero-vision SFT" approach, where only text supervised fine-tuning data is used, the Kimi K2.5 model achieved near state-of-the-art performance on vision tasks.
The Attention Residue architecture improves token efficiency by 24% according to scaling law analysis.
The Attention Residue architecture shows the most significant performance improvements on coding, math, and reasoning-heavy tasks, as measured by the GPQA, MATH, and HumanEval benchmarks.
Open models are closing the performance gap with proprietary models and are reaching the frontier of AI capabilities.
Increasing a model's context length improves its accuracy in predicting the token loss at a given position.
The agent swarms paradigm orchestrates multiple agents to accomplish subtasks in parallel, enabling the completion of more complex tasks than a single agent can handle.