Keep pulling the thread on François Chollet.
Despite a 50,000x increase in scale, LLM accuracy on the ARC-1 benchmark only improved from 0% to approximately 10%.
The AI research community began a significant pivot towards the test-time adaptation (TTA) paradigm in 2024.
A version of OpenAI's O3 model, fine-tuned on the ARC benchmark, achieved human-level performance on the benchmark for the first time in December 2024.
As of 2025, the dominant AI research paradigm has shifted from pre-training and scaling to test-time adaptation.
Fluid intelligence does not emerge from scaling up the pre-training of language models.
Test-time adaptation is a necessary component for an AI system to demonstrate genuine fluid intelligence.
Base LLMs such as GPT-4.5 and Llama 4 score 0% on the ArcGI 2 benchmark.
The ArcAGI 3 benchmark will assess agentic capabilities like exploration, interactive learning, and autonomous goal achievement.
Deep learning models lack compositional generalization, a capability the ArcGI 2 benchmark is designed to measure.
Transformer architectures are effective for Type 1 (perceptual, intuitive) abstraction but are not well-suited for Type 2 (programmatic, reasoning) abstraction.
The proposed path to advanced AI involves using deep learning's fast, approximate judgments to guide discrete program search and make it computationally tractable.
François Chollet's new research lab, Endia, is building an AI system that combines deep learning-guided discrete program search with an evolving global library of abstractions.