Post

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

Concrete example: imagine a Bucket resource creates/configures buckets on s3 API. Someone goes to the s3 console and edits bucket configuration. Unless your controller periodically queries s3 API, it won't overwrite the out-of-band modifications until next time Bucket resource is modified.

I'd expect that they're periodically querying the S3 API in this case. At a high-level, operators are supposed to continuously reconcile the current and desired states (which imply watching out for drift). Unfortunately, most external systems lack APIs to listen for changes, hence the polling.

If your controller is managing 20,000 buckets, you'll make 20k external API calls during controller startup full sync to verify buckets exist and their config is up-to-date (99.99% are) –during which you won't get to processing a new object soon enough, bc the workqueue is still full.

Same in k8s with load-balancers. If you have 100 services with LBs, and someone modifies one behind the scenes, we don't know to reconcile, because (AFAICT) none of the clouds have watch APIs and they all have API quota. Polling 100 LBs every 10 seconds = 10 QPS of 99.999% do-nothing traffic.

OTOH, maybe Kubernetes will prompt cloud providers to support some form of WATCH operations, just to alleviate the load of millions of kube clusters polling? I'm an optimist. :)