“The primary advantage of hybrid models mixing attention and recurrent layers is to obtain more expressive models that scale better during pretraining, not just to reduce memory during inference.”