Keep pulling the thread on Lilian Weng.
OpenAI trained a control policy entirely in simulation to control a human-like robot hand for solving a Rubik's Cube.
OpenAI's goal-conditioned policy for tabletop rearrangement demonstrated zero-shot generalization to unseen tasks, including those with unseen objects and configurations.
OpenAI developed Automatic Domain Randomization (ADR) to learn optimal randomization ranges during training, removing the dependency on physical robots for tuning.
The Rubik's Cube-solving policy was robust to physical perturbations not present in training, including wearing a rubber glove, having two fingers tied together, and being poked by a toy giraffe.
The "Alex's Behavior Cloning" (ABC) component was crucial for the asymmetric self-play training; disabling it completely prevented the policy from learning.
Meta-learning was a key factor in the success of the Sim2Real transfer for OpenAI's Rubik's Cube solving robot hand.
The simulation-to-real (Sim2Real) approach can fail drastically if a policy overfits to the simulation, as there are large, unmodeled differences between simulation and the real world.
OpenAI's robotic hand took approximately 4 minutes to solve a fully scrambled Rubik's Cube.
During physical trials of the Rubik's Cube-solving robot hand, the first few actions were often the slowest and most unstable, suggesting the policy was adapting to the real-world environment.
The combination of a policy with memory, such as an LSTM, and a diverse training distribution can lead to emergent meta-learning, enabling rapid adaptation to unknown dynamics.
For tasks with the complexity of solving a Rubik's Cube, standard domain randomization with fixed, predefined parameter ranges is not sufficient for successful Sim2Real transfer.
Automatic Domain Randomization (ADR) implements a training curriculum that gradually expands the distribution over environments for which the model can perform well.