Keep pulling the thread on Jeff Dean.
Major technology companies including Google, Amazon, Meta, and OpenAI are all developing their own custom AI hardware.
Google's largest Gemini models are trained using a single Python process that drives tens of thousands of chips via the Pathways system.
Google announced at Cloud Next that its Pathways system will be made available to cloud customers.
In the progression of Google's Gemini models, algorithmic improvements have been as important, or even more so, than hardware improvements or increases in compute scale.
Google's early work on mixture-of-experts models, which used 2,048 experts, demonstrated a 10x to 100x improvement in training efficiency.
In 2012, Google trained a neural network that was 60 times larger than any other at the time, using 16,000 CPU cores.
Google's TPU program was started in 2013 to address the anticipated need for a large amount of compute for inference.
Google's upcoming TPU models are named Ironwood and Trillium.
Geoff Hinton and his colleagues won a Nobel Prize in physics.
Demis Hassabis and John Jumper won a Nobel Prize for their work.
Google developed a system called Pathways, starting in 2018, to provide a better abstraction for programming across many different computing devices.
Google's Pathways system supports both PyTorch and JAX, with JAX being the primary framework used internally at Google.