Mean Of The Nines

A load generator runs 16 worker threads and reports the fleet's tail latency like this:

per_thread_p99 = []
for t in threads:                     # 16 threads
    per_thread_p99.append(percentile(t.samples, 99))

print("fleet p99 =", sum(per_thread_p99) / len(per_thread_p99))

The benchmark reports a p99 of 84 ms. Production, on the same build and the same load, shows a p99 of 310 ms. What is wrong with the benchmark?

Each thread should be weighted by its sample count instead of counting equally in the average.

It averages sixteen separate p99 values. A percentile of a merged sample is not the mean of the per-slice percentiles, so the raw samples or histograms have to be merged before the percentile is taken.

99 is not deep enough for a fleet this size. Reporting p99.9 per thread and averaging would match production.

percentile requires a sorted input and the samples are in arrival order, so each per-thread value is meaningless.