Question 869

The Sink That Never Drains

StreamingLLM found that evicting the earliest tokens from a sliding-window KV cache causes a sharp perplexity spike, even though those tokens carry little semantic content. What is the accepted explanation, and what fix restores stable long-stream generation?

A

The KV cache eviction triggers a re-tokenization of the remaining window, introducing subword boundary errors that compound over the stream

B

Softmax attention allocates a large fraction of probability mass to the first few tokens as a dumping ground for unneeded attention weight; keeping those initial tokens as permanent 'attention sink' entries in the cache while sliding the rest fixes the spike

C

The first tokens store the model's positional bias table, so removing them corrupts RoPE rotations for every later token

D

Early tokens contain the BOS embedding which is required as a normalization constant in every softmax, so its removal changes the denominator globally