Fifty Thousand Writers

A metrics ingestion service takes about 50,000 writes per second from agents, while a handful of dashboards read from it a few times a minute. The team added two read replicas and a cache, and ingestion latency did not improve. What did they get wrong in scoping?