Metrics That Never Stop

Design a metrics platform that ingests 10 million time series points per second, evaluates alerting rules on them, and serves queries over 13 months of history.

Requirements

Functional:

• Accept pushed samples from agents running on every host and container, batched over HTTP.
• Store each sample with arbitrary key/value labels, and let users query by label matchers over a time range.
• Evaluate user-defined alerting rules on a fixed interval and dispatch notifications.
• Keep 13 months of history at reduced resolution, and let one query span raw and rolled up data without the user knowing where the bytes came from.

Non-functional:

• Ingest is the availability critical path. It must stay writable when the query tier is down.
• p99 under 2 seconds for a 6 hour range query touching 10,000 series.
• Alert fires within 60 seconds of the sample that trips it.
• No sample loss on single node failure. Brief duplicates are acceptable.

Scale

• 60,000 hosts, roughly 2,500 active series each, scraped every 15 seconds.
• 150 million active series, of which about 20 percent churn daily as pods restart.
• 10 million samples per second sustained, 15 million during a fleet wide deploy.
• Raw sample is 16 bytes on the wire (8 byte timestamp, 8 byte float) before compression, plus labels.
• 2,000 dashboard queries per second, and 50,000 alerting rules evaluated every 30 seconds.
• Retention tiers: 30 days raw, 90 days at 5 minute resolution, 13 months at 1 hour resolution.

Out of scope

Logs, traces, continuous profiling, how the agent discovers what to scrape, and billing.