The formation of induction heads during training creates a visible bump in the loss curve of ever..., Sonic AI
“The formation of induction heads during training creates a visible bump in the loss curve of every transformer model with more than two layers that the speaker has observed.”