Keep pulling the thread on Nathan Lambert.
Hybrid models that combine attention and recurrence can theoretically express tasks, such as code execution, that are beyond the capabilities of pure transformers or linear RNNs.
The Olmo Hybrid model outperforms the Olmo 3 model across standard pretraining and mid-training evaluations.
The Olmo Hybrid model scales significantly more efficiently during pretraining than the pure transformer-based Olmo 3 model.
The primary advantage of hybrid models mixing attention and recurrent layers is to obtain more expressive models that scale better during pretraining, not just to reduce memory during inference.
Olmo Hybrid is a 7-billion-parameter model where the sliding window attention layers of the Olmo 3 7B model have been replaced with Gated DeltaNet layers.