Smooth Before You Round

For W8A8 post-training quantization, SmoothQuant applies a static per-channel rescaling before calibration: it divides each activation channel by $s_j$ and multiplies the matching weight channel by $s_j$ , with $s_j = \dfrac{\max|X_j|^\alpha}{\max|W_j|^{1-\alpha}}$ , mathematically preserving $XW$ . GPTQ separately quantizes weights column-by-column, using the calibration-data Hessian (input covariance) to update the remaining unquantized weights and compensate for each column's rounding error. Why do production W8A8 pipelines typically chain both rather than picking just one?

SmoothQuant quantizes the weight tensor and GPTQ quantizes the activation tensor, so chaining them is the only way to cover both operands of the matmul.

They fix different bottlenecks: SmoothQuant shrinks activation outlier channels into the weights so per-tensor int8 activation quantization stops clipping, while GPTQ then handles the residual weight-rounding error introduced by that reweighted matrix using Hessian-guided compensation. The activation-outlier problem and the weight-rounding-error problem are orthogonal.

SmoothQuant and GPTQ both implement the same Hessian-based rounding procedure, so running them in sequence just repeats the calibration pass for extra accuracy.

GPTQ needs no calibration data since it quantizes purely from the weight matrix's Frobenius norm, making it redundant with SmoothQuant, which is why they are rarely combined in practice.