Keep pulling the thread on Oriol Vinyals.
Within 5 to 10 years, it is possible there will be a pre-trained AI system that can be taught to play a complex game like StarCraft through interactive prompting, rather than specialized training.
A main challenge of the current Transformer architecture is its limited context length, which is insufficient for teaching complex tasks like StarCraft that require much larger contexts.
AlphaCode achieved human-level performance in coding competitions by combining large-scale models with a massive amount of search.
Oriol Vinyals is convinced that an AI system with human-level intelligence will be built within his lifetime.
Current large models are trained as passive observers of offline data and do not have a 'lifetime' of their own experiences.
Deployed AI models like AlphaGo, AlphaStar, and current language models do not continue to train or update their weights during interaction with humans.
A fundamental issue in AI development is the practice of training models from scratch for each new task rather than building upon previous iterations.
GPT-3 demonstrated the concept of meta-learning in the language domain, where a single, pre-trained model can be taught new tasks through prompting without retraining.
Reusing the weights of a trained neural network to expand it into a larger model is an extraordinarily hard research problem.
The Flamingo model was built using a modular approach where the weights of a pre-trained, 70-billion-parameter language model, Chinchilla, were frozen.
The largest version of the Flamingo model has a total of 80 billion parameters, consisting of a frozen 70-billion-parameter Chinchilla model and an additional 10 billion trainable parameters for vision.
The Flamingo model demonstrated an ability to converse about the subtle, joke-like context in an image of Barack Obama, a task previously highlighted by Andrej Karpathy as a challenge for computer vision systems.