“A reward model was found to resist human preference on over 25% of the HHH-RLHF dataset.”