“The preference data collection for Llama 2 likely cost between $6 to $8 million, a cost comparable to the compute used for training runs.”