Inference 0/0

Inference

Attention internals

Attention as it runs during generation: causal masks, rotary position embeddings, grouped-query and multi-head latent attention, and stretching a model to longer contexts. Each choice changes how much memory every generated token costs.

0 / 10 solved

Reading

Papers and essays1

  1. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints(opens in a new tab)

    Ainslie et al. 2023 EMNLP 2023 25 min

    Grouped-query attention: each key and value head is shared by a group of query heads, a middle point between standard multi-head attention and multi-query attention, where every query head shares one key and value head. It also gives a recipe that converts an existing multi-head checkpoint using 5 percent of its original pre-training compute.

Links2