Keep pulling the thread on Francois Chauvard.
A standard feed-forward transformer model cannot learn to map an unsorted list to a sorted list in a single forward pass.
A transformer model with N layers cannot perform an algorithm that requires more than N sequential steps, such as sorting a list of N+1 elements.
The Hierarchical Reasoning Model (HRM), a 27-million-parameter model trained from scratch on approximately 1,000 tasks, achieved state-of-the-art results on the ArcPrize 1 and 2 benchmarks.
The 27-million-parameter Hierarchical Reasoning Model (HRM) achieved approximately 70% accuracy on ArcPrize 1, while GPT-3 scored 0% on the same benchmark.
A key innovation in the Hierarchical Reasoning Model (HRM) paper is a training method inspired by Deep Equilibrium (DEQ) models that avoids full backpropagation through time.
The Tiny Recursive Models (TRM) paper shows that backpropagating through an entire latent recursion step, a deeper approach than in HRM, significantly improves model performance.
Analysis by Constantine at François Challet's company indicates that the "outer refinement loop" is the primary reason for the high performance of Hierarchical Reasoning Models (HRM).
The Tiny Recursive Model (TRM) has 7 million parameters, a reduction from the 28 million parameters of the Hierarchical Reasoning Model (HRM).
The 7-million-parameter TRM improved performance on ARC-Prize 1 to 87%, up from the 70% achieved by the larger HRM model.
A 7-million-parameter Tiny Recursive Model can solve problems that trillion-parameter LLMs trained on the entire internet cannot.
Francois Chauvard predicts that combining the recursive reasoning of models like TRM with the broad knowledge of large-scale LLMs will lead to a significant breakthrough in AI performance.
Francois Chauvard believes a promising future research direction is to use large models to create a powerful latent representation space, and then train small, recursive models like TRM to perform reasoning within that space.