Keep pulling the thread on Ilya Sutskever.
ChatGPT was created by combining a GPT foundation model with Reinforcement Learning from Human Feedback (RLHF), an approach that evolved from OpenAI's work on Dota 2.
OpenAI completed the training for GPT-4 approximately 8 months before its public launch in March 2023.
The most important difference between ChatGPT and GPT-4 is that the GPT-4 base model predicts the next word with greater accuracy.
Unreliability, including hallucinations and unexpected mistakes, is currently the single biggest obstacle for OpenAI's neural networks to be truly useful.
The GPT-4 model can consume images as input.
On the AMC 12 math competition, GPT-4's success rate improves from a 2-20% range with text-only input to 40% when vision capabilities are added.
Significant improvements in the reliability and intent-following capabilities of OpenAI's models will be seen in the next two years.
The amount of data used for training state-of-the-art neural networks increased by a factor of approximately one million between AlexNet and GPT-4.
Ilya Sutskever co-invented the AlexNet neural network with Alex Krizhevsky and Geoff Hinton at the University of Toronto.
The invention of AlexNet led to the "Big Bang" of modern artificial intelligence.
A breakthrough optimization method developed by James Martens in Geoff Hinton's lab proved that large neural networks could be trained effectively.
Alex Krizhevsky programmed the fast convolutional kernels on GPUs that were used to train the AlexNet model on the ImageNet dataset.