Keep pulling the thread on Jeff Dean.
An early large-scale unsupervised learning model at Google, trained on 10 million random YouTube frames, developed neurons that were sensitive to high-level concepts like cat faces and human faces without explicit labels.
By using an unsupervised pre-trained model for initialization, Google achieved a 70% relative improvement on the state-of-the-art for the 22,000-category version of ImageNet.
A new speech recognition model at Google halved the error rate but would have required doubling Google's total compute capacity to serve one billion users for three minutes per day.
The high computational cost of deploying a new speech recognition model convinced Google to pursue the development of specialized hardware for neural network inference and training, leading to the Tensor Processing Unit (TPU) program.
Google's TPUv1 was 15 to 30 times faster and 30 to 80 times more energy-efficient than contemporary CPUs and GPUs for neural network inference.
Google's 6th generation TPU, codenamed Ironwood, delivers approximately 3,600 times the performance per pod compared to the TPUv2.
Google's 6th generation TPU, codenamed Ironwood, is approximately 30 times more energy-efficient (flops per watt) than the TPUv2.
Google's use of sparse model architectures provides an 8x reduction in training compute for the same level of accuracy compared to dense models.
The combination of JAX and Pathways allows Google to drive an entire large-scale model training process, distributed across thousands of TPU chips, from a single Python process on one host machine.
The Pathways system at Google can automatically detect a pod with defective hardware causing silent data corruption, evict it from a training job, and transparently replace it with a hot spare to continue the training process without interruption.
Google uses distillation to create the Gemini Flash model from the larger Gemini Pro model, resulting in a smaller model that is almost as capable as the larger one.
In July 2023, a Google AI model achieved a gold medal score at the International Mathematical Olympiad (IMO) by correctly solving 5 out of 6 problems.