AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration(opens in a new tab)
Activation-aware weight quantization: find the roughly 1 percent of weight channels that matter most by looking at activation magnitudes, and scale them up before 4-bit quantization so their rounding error shrinks.