The principles of scaling compute and data, famously successful in language modeling, are being successfully translated to protein biology. Models like ESM Cambrian demonstrate that larger datasets and compute lead to predictable, log-linear performance improvements, breaking previous plateaus and suggesting a unified scaling paradigm across domains.
Naive self-play for LLMs often fails because the model learns to generate convoluted, inelegant problems to trick its solver counterpart, leading to performance plateaus. The Self-Guided Self-Play (SGS) method addresses this by grounding problem generation in existing unsolved tasks and using a 'guide' model to reward relevance.
For conversational and voice AI, user experience is critically dependent on low latency, which traditional RAG systems compromise. The StreamRag approach demonstrates the necessity of parallelizing data retrieval with user input, processing information concurrently to minimize delays and create more natural, real-time interactions.
A new paradigm for developer productivity involves adopting an 'agentic' workflow, treating software engineering like a real-time strategy (RTS) game. This involves maximizing 'actions per minute' (tool calls), parallelizing tasks, and continuously leveraging AI agents to increase output.
The integration of AI with formal verification systems like the Lean theorem prover is creating new possibilities for both fields. This includes using AI to solve complex mathematical problems (IMO, Putnam) and using formal methods to prove the correctness of AI components, such as the equivalence of flash attention to standard attention.
Keep pulling the thread on Noam Brown.