Keep pulling the thread on Rainer Pope.
Failing to batch users together for inference can result in costs that are up to 1,000 times worse than with proper batching.
To be compute-bound rather than memory-bound, the inference batch size should be greater than approximately 300 times the model's sparsity factor.
The size of a single Mixture-of-Experts (MoE) layer is fundamentally limited by the communication bandwidth within a single GPU rack, as inter-rack communication becomes a significant bottleneck.
The scale-out network for inter-rack communication is typically about 8 times slower in bandwidth than the scale-up network (like NVLink) used for intra-rack communication.
NVIDIA's scale-up domain size has increased from 8 GPUs with Hopper, to 72 GPUs with Blackwell, and is projected to be over 500 GPUs with the Rubin architecture.
An NVIDIA Blackwell rack provides a scale-up domain with 10-20 terabytes of total memory, which is sufficient to hold a 5 trillion parameter model and its KV cache.
Dwarkesh Patel cites Dylan Patel as stating that hyperscalers are spending 50% of their capital expenditures on memory in the current year.
Increasing the number of pipeline stages reduces the per-GPU memory footprint for model weights, but it does not reduce the per-GPU memory footprint for the KV cache.
The primary benefit of larger scale-up domains, such as NVIDIA's 8x increase from Hopper to Blackwell, is the aggregate memory bandwidth, not the total memory capacity.
Some papers from DeepSeek on their sparse attention mechanism show that it can change the memory fetch time calculation to scale with the square root of the context length, rather than linearly.
Rainer Pope estimates that current frontier models are over-trained by a factor of approximately 100 relative to the optimal token-to-parameter ratio suggested by Chinchilla scaling laws.
Dario Amodei has publicly stated the belief that achieving AGI will not require continual learning, as in-context learning with a sufficiently large context window will be adequate.