“The KL constraint in RLHF serves as a guardrail to prevent the model from changing too much from its base version by comparing the log probabilities of token generation between the two models.”