“A standard Transformer using Flash Attention was the first to achieve better-than-random performance on the Path-X task from the Long Range Arena benchmark, which has a 16K sequence length.”