“The performance of a sequence model is positively correlated with the size of its state; Transformers benefit from an increasing state size via the KV cache, whereas traditional RNNs are limited by a fixed-size state.”