Fifty Thousand Writers

A metrics ingestion service takes about 50,000 writes per second from agents, while a handful of dashboards read from it a few times a minute. The team added two read replicas and a cache, and ingestion latency did not improve. What did they get wrong in scoping?

The dashboards are still reading from the primary, and repointing those few queries a minute at the replicas would free the primary to absorb the agent writes

They read the workload as read heavy, but the ratio is inverted, so relief has to come from the write path (batching, partitioning, an append-optimized store)

Replicas only absorb load once the cache hit rate crosses a threshold, so a longer TTL on the dashboard queries would let the two replicas start taking work off the primary

Two replicas is too few to matter at 50,000 writes per second, so the same fan-out carried to twenty or thirty replicas would have pulled ingestion latency down