Keep pulling the thread on John Schulman.
Thinking Machines plans to release its own proprietary models in the next year.
It is possible to create a model equivalent to ChatGPT with significantly less compute by utilizing more clever algorithmic tricks.
Effective post-training techniques can substantially increase the effective compute of a model.
Extensive fine-tuning with a cleverly constructed dataset can allow a much smaller model to achieve high performance, comparable to a larger few-shot model like GPT-3.
With full hindsight, a small team of talented people could have developed a model at the level of GPT-3.5 as early as 2018 or 2019.
An early OpenAI project called Universe aimed to create a general RL agent by training on a large dataset of diverse environments like video games and web navigation tasks.
The core idea behind OpenAI's Universe project was correct but attempted approximately a decade too early due to missing prerequisites.
The OpenAI Universe project was unsuccessful at the time because its system was unwieldy for RL experiments and its models, trained from scratch, failed to generalize well.
OpenAI's robotics research was a dead end for the company in terms of direct outcomes but was valuable for building its capacity to execute large-scale engineering projects.
The Dota project was OpenAI's earliest highly successful large-scale project that utilized a significant amount of compute.
The early days of OpenAI were like "peacetime," which allowed for more exploratory research, whereas many newer AI companies must operate in "catch-up mode" to replicate the state of the art.
Value functions are not widely used in current reinforcement learning applications like RLHF because they do not provide significant benefits for tasks with verifiable rewards.