Current AI models, including large language models, have fundamental architectural limitations, failing at basic algorithmic tasks like addition with a 'carry' operation, which cannot be solved by scale or tool-use alone.
Geometric Deep Learning (GDL) successfully uses symmetry and equivariance to make models like Transformers data-efficient, but it is limited by its reliance on invertible transformations, which fails to model many real-world algorithms.
Categorical Deep Learning (CDL) is proposed as a more general and powerful framework that extends GDL.
It uses category theory to model non-invertible and compositional computations, providing a more robust mathematical foundation for neural networks.
A key missing component in models like Graph Neural Networks (GNNs) is the 'carry' mechanism.
CDL and related mathematical concepts like the Hopf fibration may provide a path to building this capability, enabling neural networks to function more like CPUs.
12 quotes
Concerns Raised
Current LLMs cannot perform reliable algorithmic reasoning, despite their massive scale.
Relying on external tools is an inefficient and brittle patch for fundamental architectural deficiencies.
Geometric Deep Learning is too restrictive because its assumption of invertibility doesn't apply to most computational algorithms.
Current neural network designs, particularly GNNs, lack a core computational primitive: the 'carry' operation.
Opportunities Identified
Develop novel architectures based on Categorical Deep Learning to overcome the limitations of current models.
Build models with intrinsic, robust, and generalizable algorithmic reasoning capabilities.
Create a unified mathematical framework for deep learning that connects high-level constraints with practical implementations.
Incorporate complex geometric and algebraic structures (like the Hopf fibration) to enable new computational primitives in neural networks.