“There are rumors that PPO-like reinforcement learning methods can achieve a higher peak performance for language models than Direct Preference Optimization (DPO).”