Every Two Minutes

Discord's Read States service tracks, for every user and channel, how far that user has read. It is hit on every connect, every message send and every message read. The Go version kept an in-memory LRU holding tens of millions of read states and absorbed hundreds of thousands of updates per second.

Its p99 showed a spike roughly every two minutes even on a flat traffic graph. The team then cut the LRU from tens of millions of entries down to a small fraction of that. The periodic spikes got smaller, and overall p99 got worse.

Which single explanation accounts for both results?

The process was large enough for the host to page part of the cache out. Shrinking it stopped the paging but lowered the hit rate.

Go forces a collection at least every two minutes regardless of heap growth, and the mark phase walks every live entry in the LRU, so the pause scales with cache size. A smaller cache marks faster but misses more, and the misses become database reads.

A collector only runs when the heap grows, so the two minute rhythm must be a periodic flush of dirty entries to the database. A smaller cache flushes less per round but misses more often.

The spike is lock contention on the cache: a smaller map has a shorter critical section, but it also holds fewer hot keys.