6 / 6 Stream Processing, Windows & Watermarks

Fast Now Correct Later

An ops dashboard shows revenue per hour. The people watching it need a usable number within about 10 seconds of the hour starting to fill, and the same hour has to be correct once stragglers land, sometimes 20 minutes late. The dashboard displays one value per hour and replaces it when a new one arrives.

Which output strategy fits, and what is wrong with the alternatives?

A

Run the streaming job for the fast approximate number and a nightly batch job over the same data for the authoritative number, letting the dashboard prefer the batch value once it exists

B

Trigger repeatedly on a processing-time period while the hour is open, again when the watermark passes the hour end, and again on each late element, with accumulating refinements so every pane carries the running total the dashboard can overwrite

C

Use the same triggers with discarding refinements so each pane carries only the delta since the last one, which is the cheaper mode and keeps less state in the operator

D

Tighten the out-of-orderness bound to 1 second so the watermark reaches the hour end almost immediately, and accept the stragglers that fall behind it as an acceptable error rate