“The KL constraint is a critical component of RLHF, and removing it causes the training process to fail.”