arXiv Notify me• Apr 21, 2026paper

SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

Tri Dao(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Tri Dao.

LLM Serving Bottlenecks System-Aware Co-design

0 quotes

Concerns Raised

Many existing KV-cache compression methods are ineffective in real-world deployment due to incompatibility with serving system constraints.
Naive INT4 quantization leads to significant accuracy degradation in LLMs.
More complex quantization methods (e.g., vector, Hessian-aware) offer marginal gains when adapted for serving compatibility.

Opportunities Identified

Significantly reduce the memory footprint of LLM serving by implementing system-aware 4-bit KV-cache quantization.
Increase LLM throughput and support more concurrent users on the same hardware without sacrificing accuracy or latency.
Achieve near-lossless compression with zero performance overhead by using fused kernels for quantization.

Key Themes

Research Findings7

The SAW-INT4 method, which uses token-wise INT4 quantization with block-diagonal Hadamard rotation, consistently achieves the best accuracy-efficiency trade-off for 4-bit KV-cache quantization under practical serving constraints.

The SAW-INT4 method for KV-cache quantization recovers nearly all of the accuracy lost by naive INT4 quantization.

The fused rotation-quantization kernel for the SAW-INT4 method introduces zero measurable end-to-end overhead.

The SAW-INT4 KV-cache quantization method matches the throughput of plain INT4 quantization across all concurrency levels.

Many existing KV-cache compression methods are ineffective in deployment as they violate practical serving constraints like paged memory layouts, regular memory access, and fused attention execution.

More complex methods for KV-cache compression, such as vector quantization and Hessian-aware quantization, provide only marginal additional gains over SAW-INT4 once serving compatibility is considered.

A fused rotation-quantization kernel for the SAW-INT4 method was implemented to integrate directly into paged KV-cache layouts.

Topics

LLM Serving KV-Cache Quantization INT4 Memory Optimization Model Compression Inference Efficiency Throughput Latency Systems Co-design Fused Kernels Paged Attention Hadamard Rotation AI Infrastructure Large Language Models

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

LLM Serving Bottlenecks

System-Aware Co-design

Practical Quantization Techniques

Fused Kernels for Zero-Overhead Implementation

Research Findings7

Topics