Keep pulling the thread on Jim Fan.
In the Voyager project, GPT-4 demonstrated in-context lifelong learning by committing successful code to a memory repository and retrieving it for similar future situations, all without fine-tuning its parameters.
Project Groot is NVIDIA's moonshot initiative to solve general-purpose robot AI.
The primary challenge in robotics is the data problem, as there is no large, pre-existing dataset equivalent to the internet for training robot control policies.
NVIDIA uses domain randomization, varying physical parameters like gravity and friction in simulations, to enable zero-shot transfer of learned policies from simulation to the real world.
NVIDIA can provide humanoid robots with 10 years' worth of training in only 2 hours of simulation time.
The Groot Dreams algorithm fine-tunes a pre-trained video generation model on robot data to create a world model capable of simulating robot behavior and generating synthetic training data.
NVIDIA has open-sourced Groot-N1, an end-to-end robotic foundation model that uses a VLM for reasoning and a diffusion model to render actions at over 100 Hz.
The number of intelligent robots in the world will be greater than the number of iPhones by 2040.
A "GPT-3 moment" for robot foundation models will likely occur by 2030.
Jim Fan co-leads the development of the Isaac Group platform for humanoid robotics at NVIDIA.
Jim Fan's past work includes Voyager, the first AI agent to master Minecraft through lifelong learning.
The 2016 OpenAI project World of Bits aimed to teach an AI agent to use a web browser by observing pixels and outputting keyboard and mouse controls.