No Peeking Ahead

A decoder-only transformer generates text left to right. Why does self-attention in this decoder apply a causal mask to the $QK^\top$ score matrix before the softmax?

It randomly drops a fraction of attention scores each forward pass to regularize training

It sets scores for future (not-yet-generated) positions to negative infinity so softmax assigns them zero weight, preventing a token from attending to tokens after it

It zeroes out the diagonal of the score matrix so a token cannot attend to itself

It scales down scores for distant tokens so nearby tokens always dominate the output