Counted Once In Pinot

Uber's ad event pipeline is a chain of Flink jobs over Kafka. Flink runs with exactly-once checkpointing every two minutes, the Kafka producers are transactional, and downstream consumers read with read_committed. The aggregation job emits one-minute tumbling windows keyed by ad and minute; the attribution job joins those to orders and writes to Pinot and Hive. Aggregation results carry a random UUID, attribution results carry a deterministic id.

A task manager dies. Flink restores the last checkpoint and reprocesses about two minutes of events. Pinot totals stay correct.

Which combination is doing the work?

The random UUID on each aggregation result makes every row unique, which is what lets the sink detect and drop repeats.

The Kafka transaction opened by the failed attempt is aborted, so read_committed consumers never observed its output, and the replayed attribution rows carry the same deterministic id, so Pinot upserts over the earlier row instead of adding one.

Flink's exactly-once checkpointing covers the whole path, so any system reading the output is automatically correct.

Pinot discards rows whose event time falls inside a window it has already sealed, using the two minute checkpoint interval as the cutoff.