Keep pulling the thread on Jan Leike.
In experiments with GPT-4 class models, weak-to-strong generalization shows non-trivial positive results on a range of NLP tasks, with performance increasing even when extrapolating across four orders of magnitude in model size difference.
Weak-to-strong generalization does not work as well on reward modeling tasks as it does on general NLP tasks.
On reward modeling tasks, the generalization lines for weak-to-strong learning are mostly flat, indicating the strong student model does not significantly outperform the weak teacher model.
The poor performance of weak-to-strong generalization on reward modeling tasks is evidence that Reinforcement Learning from Human Feedback (RLHF) will not scale.
In the "Critiques" project, models trained for helpfulness using RLHF also became significantly better at finding bugs in code, with an improvement equivalent to a 30x increase in pre-training compute.
The primary task for the AI safety field should be to align an automated alignment researcher, which can then be used to bootstrap further alignment progress.
Solving the problem of fully eliciting a model's capabilities would also largely solve the problem of AI deception.
Currently, humans evaluating responses from models like ChatGPT or Claude are not yet considered "weak supervisors" because they can generally determine the quality of the output.
Experiments testing whether AI "Critic" model assistance improves human accuracy at finding bugs in code were inconclusive and not statistically significant.
On the GSM dataset, "checkability training" with small models successfully transferred to human evaluators, increasing their accuracy and decreasing the time they took to judge solutions.
Standard RL training for correctness on the GSM dataset caused human judge accuracy to decrease over time, in contrast to the improvement seen with "checkability training".
Evaluating AI-generated alignment research is easier than generating that research from scratch.