The Average Of Percentiles

Forty instances export latency. Two dashboard panels claim to show the fleet's p99:

# panel A
avg(rpc_latency_summary_seconds{quantile="0.99"})

# panel B
histogram_quantile(0.99, sum by (le) (rate(rpc_latency_seconds_bucket[5m])))

Right now 39 instances report a p99 of 20 ms. One instance is wedged: its own p99 is 4,000 ms, and it is taking 2 percent of fleet traffic with essentially all of its requests slow.

What do the two panels show, and which one should page you?

Panel A is the correct one; panel B double counts, because sum over rate adds the same request once per bucket boundary it falls under.

Panel A reads $(39 \times 20 + 4000)/40 = 119.5$ ms and is meaningless, because quantiles do not average. Panel B is the real fleet p99, and since the wedged instance carries 2 percent of traffic its slow requests sit above the 99th percentile line, so panel B reads in the seconds.

Both are valid estimators of the same quantity; panel A is simply noisier because it weights every instance equally regardless of traffic.

Panel A reads 119.5 ms and panel B reads about 20 ms, because 39 of the 40 instances are fast and the summed histogram is dominated by them.