Post

A very cool FLIP (Flink Improvement Proposal) has recently been accepted: Running Flink jobs in MiniCluster using the Kubernetes Operator. cwiki.apache.org/confluence/s...

Historically, every Flink pipeline required: - At least two pods (JobManager and TaskManager). - Each pod needs at least 0.5 vCPU and 1GB RAM (it's possible but really tricky to go lower).

So that's 1 vCPU and 2GB RAM for any pipeline to start, even if it processes one message a minute. Imagine running that at scale (I don't need to imagine...)? At the same time, other tools (e.g., Rust- or Go-based) can have a much lower footprint.

The FLIP addresses the problem by running both components (JobManager and TaskManager) in the same JVM process on a single pod. It's a big step in making Flink simpler and more efficient to operate.