Big Disk Slow Pipe

An Open Connect site holds two generations of appliance in one cluster:

Type	Storage	Throughput
Disk server	about 200 TB	about 40 Gbps
Flash server	up to 18 TB	up to 100 Gbps

Content is placed on a consistent hash ring, each server taking about 1000 positions, weighted by storage. Requests follow the content: a server is asked for exactly the titles it holds.

What breaks, and what does Netflix do about it?

Store the full catalog on both types and let the load balancer split requests in proportion to throughput.

Weighting by storage hands the flash box a sliver of the catalog and therefore a sliver of the requests, while it can serve more than twice the disk box's throughput. Netflix splits the catalog by popularity across two weighted rings.

Weight the single ring by throughput instead of storage. Give the flash server two and a half times the positions and traffic matches capacity.

Increase the number of hash positions per server so the distribution smooths out.