4 / 9 Quantization & Low Precision

Salient Channel Rescue

AWQ (Activation-aware Weight Quantization) decides which weight channels to protect from quantization error based on:

A

Random sampling of channels to keep protection unbiased

B

The row norm of the layer's Hessian matrix

C

The raw magnitude of the weights themselves

D

The magnitude of the activations multiplying each weight channel, since large activations amplify quantization error on that channel

E

Gradient magnitude collected during a short fine-tuning pass