Rolled Up Too Far

Latency samples arrive every 10 seconds and are rolled up into hourly averages, with the raw samples expiring after 14 days. A p99 latency dashboard looks correct for the last two weeks, then flattens into a much lower line for everything older. What explains it?

The hourly points are sparser than the dashboard's step, so the graph interpolates between neighboring points and smooths the spikes away

Counters reset whenever a process restarts, so every deploy inside the older window pulls the historical p99 line down toward zero

Quantiles cannot be reconstructed from stored per-interval means, so once raw samples expire the rollup can only report the aggregate it precomputed

Clock skew between collectors places older samples into neighboring hour buckets, which spreads the tail across several of them and lowers the line the tail draws