Two Thousand Logical Shards

Instagram runs several thousand logical shards on a much smaller number of physical Postgres machines. Each logical shard is a schema (insta5, insta6, and so on) with its own tables and its own next_id function, and the map from logical shard to machine lives in application code.

Machine	Logical shards	Disk used
pg-a	0 to 511	91%
pg-b	512 to 1023	46%
pg-c	1024 to 1535	44%
pg-d	1536 to 2047	42%

pg-a is nearly full and its CPU is climbing. What does this layout let the team do about pg-a that a straight hash of user id to machine would not?

Add read replicas and accept that the write hotspot is permanent, because the shard id is baked into every id already issued.

Copy the schemas for logical shards 256 through 511 to a new machine and flip those entries in the map. Every id already names its logical shard, so no row is re-hashed.

Rehash every row into a larger keyspace, which is cheap because the shard count was chosen to be a multiple of the machine count.

Widen the 13 bit shard field so more shards exist, then redistribute.