The Modulo Trap

Your cache tier shards keys with hash(key) % 8 across eight servers. You add a ninth server and the cluster-wide cache hit rate falls to nearly zero within seconds. What explains the collapse?

Adding a server widens the hash function's output range, so every key now produces a different hash value than it produced before

Changing the divisor from 8 to 9 reassigns almost every key, so roughly eight ninths of the keys now resolve to a server that never stored them

Only the keys that lived on the busiest of the eight servers were reassigned, and that one server happened to be serving most of the read traffic

The ninth server starts out empty and must warm up from the origin, while the original eight keep serving all of the keys they already hold