Keep pulling the thread on Aditi Raghunathan.
Many academics have found that Llama 3 is significantly harder to fine-tune compared to its predecessor, Llama 2.
The difficulty in fine-tuning Llama 3 is attributed to it being trained on an excessive amount of data, which improves benchmark performance but makes it a worse starting point for custom tasks.
Retrieval-Augmented Generation (RAG) is not a complete solution for keeping models updated, as models often fail to use retrieved context to override their internal parametric knowledge.
Experiments on Olmo checkpoints revealed that the 1-billion-parameter model trained on 3 trillion tokens performs worse after fine-tuning than a version trained on fewer tokens.
Standard LLM training methods do not naturally encourage information to be localized in specific neurons, making it fundamentally difficult to "unlearn" or remove specific knowledge post-training.
The "memorization sinks" technique trains a model by designating specific neurons that are updated only on a particular document, isolating that document's information to enable easier unlearning.
Multi-token prediction, which forces a model to generate an entire sequence at once without intermediate feedback, encourages a more global understanding and can improve model creativity.
Diffusion models can improve creativity and global planning ability because their training process involves masking and varied token ordering, which prevents reliance on local patterns.
A more effective method for generating structured, diverse outputs than temperature sampling is to have the model first commit to a random idea or prefix and then generate the full response based on that starting point.
Despite GPT-5's high benchmark scores, which suggest it is the "smartest model around," the user experience for many has been lacking.
For model quantization, research found that performance degradation from overtraining is directly proportional to the ratio of training tokens to model parameters.
A 1-billion-parameter model trained on more than 2.5 trillion tokens is likely a poor choice for fine-tuning unless the target task is very close to the pre-training data.