5 / 9 Quantization & Low Precision

Skinny Weights, Fat Activations

For memory-bound LLM inference at small batch sizes, weight-only quantization (e.g. INT4 weights with bf16 activations) is often preferred over full INT8 weight-and-activation quantization. Why?

A

Activations are always more compressible than weights because they change on every forward pass

B

Full INT8 quantization always beats weight-only quantization on both speed and accuracy for LLMs

C

Inference is bottlenecked by streaming weights from memory, so shrinking weights cuts load time, while keeping activations in higher precision avoids the accuracy hit from activation outliers that are harder to quantize safely

D

GPUs are physically incapable of multiplying two INT8 tensors together

E

Weight-only quantization removes the dequantization step from the inference path entirely