Keep pulling the thread on Lilian Weng.
OpenAI trained a robot hand control policy entirely in simulation to solve a fully scrambled Rubik's Cube on a physical robot.
OpenAI developed Automated Domain Randomization (ADR), a method that implements a training curriculum by starting with a narrow distribution of environments and gradually expanding it as the policy improves.
Erasing the LSTM hidden state of the Rubik's Cube policy during a trial caused the time-to-complete a flip to increase, indicating the policy relies on memory to adapt to the environment.
OpenAI trained a single goal-conditioned policy that can zero-shot generalize to unseen robotic manipulation tasks using an asymmetric self-play framework.
Using the goal-generator's (Alice's) trajectory as a demonstration for behavior cloning was crucial for the goal-solver (Bob) to learn; without it, the policy failed to learn.
Initial attempts to transfer a policy trained in simulation to a real robot for the Rubik's Cube task failed due to unmodeled differences between the simulation and the real world.
The Rubik's Cube-solving policy exhibited adaptive behavior where initial actions were slower and more unstable, but became faster and smoother after 5 to 10 rotations.
A policy trained with Automated Domain Randomization (ADR) achieved higher sim-to-sim transfer performance and trained faster than policies trained with fixed randomization ranges.
The accuracy of predicting environment parameters, such as cube size, from a policy's memory state showed a positive correlation with the diversity of the training environment.
OpenAI's attempts to use generative or energy-based models to learn the parameter distribution for domain randomization did not yield significant performance improvements over a simpler factorized ADR approach.
In the asymmetric self-play framework, a goal-generating policy (Alice) is rewarded when the goal-solving policy (Bob) fails, motivating it to create increasingly difficult tasks.
The policy trained for tabletop manipulation could not reliably stack more than three blocks, indicating a limitation in handling more complex, long-horizon tasks.