Post

Watermarks can be a really scary concept when you start working with stateful streaming systems. And all popular frameworks don't offer anything better than "bounded out-of-orderness": pick a number of seconds and hope that all data arrives within that interval.

But in reality, there is almost always late data, so you end up with data loss (and incorrect results). This is a really hard problem, especially when you don't control your producers (e.g. web/mobile click stream data, IoT telemetry, etc.)

But when you do control the producers (e.g. application events), the producers themselves can be the source of watermarks! The idea is not new, and it's covered in several famous papers like Google's MillWheel (see the screenshot).