Keep pulling the thread on Jeff Dean, Noam Shazeer.
An early Google Brain experiment trained a large neural network on 10 million YouTube frames using 16,000 CPU cores, resulting in the unsupervised emergence of a "cat neuron" that selectively responded to images of cats.
The value creation opportunity for AI systems that can perform tasks and solve problems at scale is orders of magnitude larger than the opportunity in organizing information, potentially reaching a quadrillion dollars.
Scaling AI models to attend to trillions of tokens is a major computational challenge because the naive quadratic attention algorithm is not feasible beyond millions of tokens.
Google has fine-tuned a Gemini model on its internal codebase for use by its own developers.
25% of the characters checked into Google's codebase are generated by its internal AI coding models.
It is plausible that within the next one to two years, AI systems will be able to automatically generate experimental code based on high-level research ideas.
An experimental Google coding model was able to generate a functional SQL processing database system in C, including a parser, tokenizer, query planner, and storage format, from a single paragraph prompt.
Modern top AI labs have approximately one million times more compute available than was used to train the original Transformer model.
Improvements in AI models from one generation to the next are driven as much, or more, by algorithmic advances and data mixture adjustments as they are by increased scale and better hardware.
A major source of AI progress in the near future will come from exploiting cost headroom to apply significantly more compute at inference time.
Language model inference is approximately 100 times cheaper than reading a paperback book, 10,000 times cheaper than a customer support agent, and over a million times cheaper than a software engineer.
Google trained its Gemini 1.5 model across multiple metro areas using high-bandwidth, long-latency connections between data centers.