Keep pulling the thread on Jay Dawani.
At frontier AI labs, GPU utilization for training workloads is typically between 45% and 50%.
For many AI companies, GPU utilization for inference workloads is often under 15%.
The gap between hardware capabilities and the software's ability to utilize them is widening with each new hardware generation.
The cost of moving data from memory to an ALU is approximately 100 times more energy-intensive than the computation itself.
Approximately 70% of a chip's total system energy is consumed by moving data, while only 10-15% is used for arithmetic operations.
The AI software industry has historically focused on optimizing the 10-15% of energy spent on arithmetic rather than the 70% spent on data movement.
Hyperscalers currently require a 1,000x increase in computational capability for roughly the same power consumption.
With a 32K token context length, a modern GPU can only fit approximately four sequences, causing Model Flop Utilization (MFU) to drop below 10%.
In exascale computing setups, 70-80% of cycles are wasted on coordination and memory overhead due to the limitations of the kernel-based programming model.
Current static software approaches result in AI systems achieving only 20-30% of their theoretical peak performance for a given workload under ideal conditions.
Jay Dawani argues that in the current AI landscape, "time to production" is the most important competitive variable for companies.
Approximately eight years ago, training a 20 billion parameter model required a few thousand GPUs and several months of training time, followed by additional months for validation.