Inference 0/0

Inference

Quantization

Storing weights, activations and the KV cache in fewer bits: bf16, FP8, INT4 weights, and methods such as GPTQ, AWQ and SmoothQuant. Fewer bytes per weight means faster memory-bound decoding and a larger model per GPU.

0 / 9 solved

Reading

Papers and essays2

  1. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration(opens in a new tab)

    Lin et al. 2024 MLSys 2024 50 min

    Activation-aware weight quantization: find the roughly 1 percent of weight channels that matter most by looking at activation magnitudes, and scale them up before 4-bit quantization so their rounding error shrinks.

  2. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers(opens in a new tab)

    Frantar et al. 2023 ICLR 2023 75 min

    GPTQ: quantize a trained model's weights to 3 or 4 bits layer by layer, using approximate second-order information to adjust the not-yet-quantized weights for each rounding error. A 175B model takes about four GPU hours.

Links1