Position Encoding Showdown

A team needs a decoder-only LLM to reliably attend across sequences much longer than anything seen in training, without retraining position embeddings. Why is ALiBi generally favored over RoPE for this specific extrapolation goal?

ALiBi removes positional information entirely, so the model has no distance bias to break when sequences get longer

ALiBi adds a fixed linear distance penalty directly to attention scores instead of rotating query/key vectors, so it degrades attention smoothly and monotonically with distance beyond the training length

ALiBi encodes absolute positions as learned embeddings, so it can be fine-tuned cheaply for any new context length

ALiBi and RoPE are mathematically identical, so the extrapolation gap is caused entirely by tokenizer differences, not position encoding