Order, Please

A vanilla transformer block (self-attention followed by a feed-forward network, each wrapped with residual connections and layer normalization) processes all tokens of a sequence in parallel with no recurrence. Why must positional encodings be added to the token embeddings before this block runs?

Positional encodings are needed to keep the attention scores from exceeding the softmax's numerical range

Self-attention is permutation-invariant, so without positional information the model cannot distinguish token order

The feed-forward sublayer requires positional input to compute its residual connection

Layer normalization only works correctly when inputs are position-tagged