“A key difference between traditional RLHF and DPO is that RLHF allows for separate training and testing of a reward model and a policy, whereas DPO combines these into a single update step.”