“The representations of BERT encoders become increasingly unexploitable by frozen probes as the models scale, despite improvements in perplexity.”