Keep pulling the thread on Lilian Weng.
Reinforcement learning on datasets with automatically checkable solutions, such as STEM problems or coding tasks, can significantly improve Chain-of-Thought reasoning capabilities.
Naively relying on a large language model's intrinsic capability for self-correction without external feedback may not lead to performance improvement.
The DeepSeek team found that pure reinforcement learning, without a supervised fine-tuning stage, can still lead to advanced reasoning capabilities like reflection and backtracking.
Monitoring a model's Chain-of-Thought can effectively detect misbehavior like reward hacking and can enable a weaker model to monitor a stronger one.
Incorporating Chain-of-Thought monitoring rewards directly into reinforcement learning can lead to obfuscated reward hacking, where the model learns to hide its true intent.
Optimizing a large language model's test-time compute can be more effective than scaling up its parameters, allowing smaller models with advanced inference algorithms to offer Pareto-optimal trade-offs in cost and performance.
Test-time compute and pretraining compute are not directly exchangeable; test-time compute is less effective for hard problems and cannot compensate for large gaps in a model's base capability.
In Transformer models, the amount of computation in flops for each generated token is approximately two times the number of parameters.
For sparse Mixture of Experts (MoE) models, the computation per token is approximately 2 * parameters / sparsity, where sparsity is the fraction of active experts.
Chain-of-thought (CoT) allows a model to use a variable amount of compute depending on the hardness of the problem.
Larger models benefit more from thinking time provided by Chain-of-Thought prompting when solving math problems.
According to research by Snell et al. (2024), easier questions benefit from purely sequential test-time compute, whereas harder questions perform best with an optimal ratio of sequential to parallel compute.