Spin to Encode

Rotary Position Embedding (RoPE) encodes token position by applying a position-dependent rotation to query and key vectors before computing $QK^\top$ . What property does this rotation give the attention scores?

It compresses the key and value vectors into half their original dimensionality for faster inference

The dot product between a rotated query and key depends only on their relative position (the difference in their positions), not their absolute positions

It guarantees attention scores are always symmetric, so token $i$ attends to token $j$ exactly as much as $j$ attends to $i$

It removes the need for the $\sqrt{d}$ scaling factor since rotation already normalizes vector magnitude