“Hybrid models that combine attention and recurrence can theoretically express tasks, such as code execution, that are beyond the capabilities of pure transformers or linear RNNs.”