Separating Representation from Reconstruction Enables Scalable Text Encoders

CrossBERT is a two-part architecture that separates the learning of encoded representations from the token reconstruction process.

The CrossBERT architecture enables high masking ratios of 50% or more, increasing training throughput by 1.5 to 2 times.

The Complementary Masking Strategy in CrossBERT increases sample efficiency by 2 times by enabling gradient collection over all tokens.

CrossBERT achieves superior performance on the MTEB (eng, v2) and frozen GLUE benchmarks.

Text encoders have remained largely unchanged and have not scaled as rapidly as decoders since the introduction of the BERT architecture.

The representations of BERT encoders become increasingly unexploitable by frozen probes as the models scale, despite improvements in perplexity.

The misalignment in BERT's representations is caused by its flat design, which couples representation learning to the token reconstruction loss.