Inference
Quantization
Storing weights, activations and the KV cache in fewer bits: bf16, FP8, INT4 weights, and methods such as GPTQ, AWQ and SmoothQuant. Fewer bytes per weight means faster memory-bound decoding and a larger model per GPU.
Questions
Question Difficulty Success