Keep pulling the thread on Janusz Maretski.
The AI industry ran out of high-quality, diverse training data approximately three and a half years ago.
Training new AI models on recent internet data, which is increasingly generated by other LLMs, leads to a phenomenon called "model collapse" where the models degrade in quality.
The performance of large language models on standard benchmarks has plateaued and has not shown significant improvement for the past three years.
Janusz Maretski predicts that within one to three years, the majority of laptops will be capable of running the most powerful general-purpose language models locally.
A fundamental limitation of the current generation of neural networks is their inability to learn continuously from user interactions.
The problems of hallucination and compounding errors in current AI models are impossible to solve using existing neural network techniques and require a fundamentally new approach.
Next-generation AI systems from companies like Fractal Brain and Innate AI achieve 3-4 orders of magnitude better power efficiency than current models.
Next-generation, continually learning AI systems are already outperforming current systems on common benchmarks.
Prominent AI researchers including Jan LeCun, Dave Silver, Ilya Sutskever, and Andrei Karpathy have left senior roles at major AI labs to pursue new approaches, signaling their belief that current LLM technology has fundamental limitations.
Dave Silver recently left DeepMind to form a new company called Ineffable Intelligence, which is reportedly raising $1 billion.
The hundreds of billions of dollars being invested in large-scale data centers for current AI models represents a "catastrophic misallocation of capital."
Janusz Maretski predicts that companies that have avoided massive capital expenditures on data centers and frontier LLM training, such as Apple, will be the long-term winners in the AI market.