Softmax on a Rotating Ring

Ring Attention partitions a long sequence's Q/K/V blocks across devices arranged in a logical ring; each device holds a fixed Q block while K/V blocks are rotated peer-to-peer around the ring, overlapping the transfer of the next block with attention compute on the current block. Since each device only ever sees one K/V block at a time and never materializes the full $N \times N$ attention matrix, what technique does it rely on to still produce the mathematically exact (not approximated or sparsified) full-sequence attention output?

A low-rank projection of each K/V block before it is sent around the ring, trading some accuracy for reduced inter-device bandwidth.

A final all-reduce over locally-computed partial logits after every device has gathered all K/V blocks onto itself.

Online (streaming) softmax with running max and running sum rescaling, so each newly arrived K/V block's partial attention contribution can be incrementally folded into a running output that converges to the exact full-sequence softmax.

Sparse attention masking that skips K/V blocks whose block-level similarity score falls below a threshold, bounding total compute.

Int8 quantization of K/V blocks prior to each hop of the ring rotation to shrink the communication payload.