Fifty Doors One Answer

A search API fans a query out to all 50 shards in parallel and merges the results. Each shard independently shows a median of 8 ms and a p99 of 40 ms, yet the end-to-end p99 measured at the coordinator is well above 100 ms. What explains the gap?

Fifty concurrent connections exhaust the coordinator's connection pool, so most requests spend their time queued

The response waits on the slowest of 50 parallel calls, so roughly 39% of requests touch at least one shard responding at or beyond its own p99

The coordinator queries shards one after another, so the per-shard latencies add up across the fan-out

Merging and re-ranking 50 partial result sets is CPU bound at the coordinator, and that merge step adds tens of milliseconds to every request it serves