“To save memory, Flash Attention does not store the large N x N attention matrix from the forward pass and instead recomputes it on-the-fly in small blocks during the backward pass.”