Inference costs dominate training costs when serving a large language model to billions of users.

The Speculative-Speculative Decoding (SSD) algorithm can correctly predict verification outcomes approximately 80% to 90% of the time.

Using the Speculative-Speculative Decoding (SSD) algorithm, it is possible to achieve a sampling speed of 300 tokens per second for Llama 3 70B on a system with four H100 GPUs.

Among the inference engines tested for the Speculative-Speculative Decoding paper, SG Lang was the fastest for implementing standard speculative decoding.

The Speculative-Speculative Decoding (SSD) algorithm predicts likely verification outcomes by using the token distributions from the draft model, specifically considering tokens that were plausible bu...

The Speculative-Speculative Decoding (SSD) algorithm provides improvements in both latency and throughput, whereas standard speculative decoding is typically only a clear win for latency.

The Speculative-Speculative Decoding (SSD) algorithm achieves additional speedups because the time taken for verification allows for drafting more tokens, which increases the expected number of accept...

Within the training process for large models, reinforcement learning (RL) is beginning to require more compute than the initial pre-training phase.

In vanilla speculative decoding, it is possible to sample an extra "bonus" token for free at the point of rejection without requiring an additional forward pass.

The speaker predicts that within one to three years, LLM inference will be viewed as a core capability that determines a model's peak intelligence, rather than just a cost or convenience factor.

The goal of the Speculative-Speculative Decoding (SSD) algorithm is to parallelize the drafting and verification steps of speculative decoding, allowing them to happen concurrently.

The transformer architecture allows for parallel verification of token probabilities in a single forward pass, whereas token generation is an autoregressive, one-at-a-time process.

The top SKU for ChatGPT is priced at $200 per month.

A speaker argues that ChatGPT possesses a business model of significantly higher quality than analogous consumer companies from previous product cycles.

The top consumer SKU offered by Google is priced at $250 per month.

AI-native alternatives to LinkedIn are being developed to create user profiles that contain a person's actual knowledge, allowing for synthetic interaction, rather than just listing skills.

Apple's AirPods are the most widely adopted consumer electronic device to be released since the smartphone.

The product Granola has experienced significant user growth because it enables people to derive value from their daily spoken conversations.

Despite perceived interchangeability, providers of large language models like Claude, ChatGPT, and Gemini are raising prices rather than lowering them.