Keep pulling the thread on Oriol Vinyals.
Google's Gemini models can process a one-hour video and answer questions about its content within 10 to 30 seconds.
The standard context length for commercially available LLMs is expected to increase by approximately 10x within the next 1-2 years.
Current state-of-the-art LLMs possess reasoning capabilities but lack the crispness and accuracy to avoid hallucinations or consistently solve basic puzzles, even when they can solve complex problems.
In current LLM training, most compute is spent on pre-training, and the reinforcement learning phase is often stopped early to prevent overfitting to imperfect reward functions.
The future allocation of compute for training general AI models will likely shift from over 90% for pre-training today to around 50%, with the remainder dedicated mostly to reinforcement learning.
The Gemini project was formed by merging two parallel LLM development efforts from Google Brain and legacy DeepMind.
Google DeepMind was formed by merging Google Brain and legacy DeepMind into a single AI research organization.
Google's Project Astra is an AI agent on a phone that uses memory to enable continuous interaction.
Hierarchical memory architectures will remain relevant for achieving longer context windows in LLMs due to efficiency benefits.
The future of information processing in LLMs will likely be a hybrid model combining direct context processing with retrieval-based methods.
A key research problem for language models is the difficulty of assigning a clear reward signal (true/false), unlike in games, because language is inherently fuzzy and subjective.
A key hypothesis for advancing AI is that it is easier for a model to verify a correct solution than to generate one, enabling models to effectively evaluate their own outputs for reinforcement learning.