“Flash Attention enables training Transformer models with sequence lengths up to 16K or 64K.”