“Weak-to-strong generalization does not work as well on reward modeling tasks as it does on general NLP tasks.”