“During preference data collection, human labelers are given a long document of instructions that specifies a hierarchy of values to prioritize, such as factuality, helpfulness, honesty, and harmlessness.”