Keep pulling the thread on Tri Dao.
ScaleSearch is a strategy for selecting scale factors in Block Floating Point formats that uses a fine-grained search leveraging mantissa bits to minimize quantization error for a given distribution.
ScaleSearchAttention is an accelerated, NVFP4-based attention algorithm that uses ScaleSearch and adapted prior techniques to ensure near-zero performance loss for causal language modeling.
The ScaleSearch method reduces quantization error by 27% for the NVFP4 format.
The ScaleSearch method improves Post Training Quantization (PTQ) performance by up to 15 points on the MATH500 benchmark for the Qwen3-8B model.
The ScaleSearchAttention algorithm improves Wikitext-2 perplexity (PPL) by up to 0.77 points for the Llama 3.1 70B model.
Using a fixed scale based on the maximum magnitude of a block in standard Block Floating Point (BFP) algorithms can be suboptimal with respect to quantization errors.
The ScaleSearch method can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention.