“Transformers are fundamentally permutation equivariant models, which makes them exponentially more data-efficient for learning symmetries compared to a simple Multi-Layer Perceptron (MLP).”