Keep pulling the thread on Multi-Layer Perceptron.
A core limitation of geometric deep learning is its assumption that all transformations are invertible, which fails to model many computational processes that destroy information.
Major scientific advances using LLMs, such as those from AlphaCode, AlphaGeometry, and FunSearch, have resulted from combining the language model with an external algorithmic procedure like a theorem prover or genetic algorithm.
The central proposal of Categorical Deep Learning is that a neural network layer should be viewed as a homomorphism between two algebras for the same endofunctor.
Current language models like ChatGPT cannot perform algorithmic addition, failing on problems that require carrying over digits, even though they can solve some arithmetic questions through pattern matching.
Physics-based AI models like V-JEPA and Genie do not yet accurately model physical principles like Newton's three laws of motion and are not reliable enough for applications in robotics.
Current frontier language models perform hundreds of billions of multiplications to produce a single output token, yet they cannot reliably multiply relatively small numbers.
Andrew from DeepMind argues that relying solely on tool use for language models is insufficient and that improving the underlying model architecture remains essential.
Transformers are fundamentally permutation equivariant models, which makes them exponentially more data-efficient for learning symmetries compared to a simple Multi-Layer Perceptron (MLP).
The research field of Categorical Deep Learning is an attempt to establish a foundational theory for neural networks, analogous to the periodic table for chemistry.
Pathfinding algorithms like Dijkstra's and Bellman-Ford are non-invertible, as different input graphs can yield the same output, making them impossible to model with the symmetry groups used in geometric deep learning.
The Categorical Deep Learning framework generalizes Geometric Deep Learning, which is recovered as the special case where the homomorphism corresponds to a group action.
The design of Graph Neural Networks (GNNs) has largely overlooked the arithmetic concept of a "carry," which is fundamental for assembling complex computations.