“The compute budget used for alignment fine-tuning of GPT-4 was a smaller fraction of its pre-training compute than the budget used for GPT-3.”