GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints(opens in a new tab)
Grouped-query attention: each key and value head is shared by a group of query heads, a middle point between standard multi-head attention and multi-query attention, where every query head shares one key and value head. It also gives a recipe that converts an existing multi-head checkpoint using 5 percent of its original pre-training compute.