Standard transformer-based LLMs process inputs in a single, parallel forward pass. This architecture makes them unable to perform algorithms that require more sequential steps than the model has layers, limiting their ability for "latent reasoning" on complex problems like sorting an unsorted list.
The episode discusses how Hierarchical Reasoning Models (HRM) and Tiny Recursive Models (TRM) successfully reintroduce recursion, a concept central to older Recurrent Neural Networks (RNNs). Unlike LLMs' "Chain of Thought" which reasons in the discrete token space, these models perform recursive operations in a continuous latent space, enabling more powerful and efficient reasoning.
The discussion highlights a stark contrast between tiny, specialized models (TRM is 7M parameters) that excel at specific reasoning tasks and massive, general-purpose LLMs (trillions of parameters) that possess broad knowledge but are less efficient at these tasks. The recursive models must be trained per-task, unlike generalist LLMs.
A major reason for the decline of RNNs was the difficulty of training them using backpropagation through time (BPTT), which led to vanishing or exploding gradients. The new models (HRM/TRM) introduce a novel training technique that only backpropagates through a single, truncated recursive step, making the training process stable and effective.
The speakers predict that the most powerful future AI systems will be hybrids. They will leverage large models to create rich, semantically meaningful latent representations of the world, and then use small, efficient recursive models like TRM to perform complex reasoning within that latent space.
Keep pulling the thread on Francois Chauvard.