Post

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

Ideally an operator would watch the resources it manages and revert drift as soon as possible, to minimize downtime. But in the case where watching is not possible, polling may be acceptable. For scaling, consider adding a component that centralizes the polling and transforms it into watches.

To convert data polled from external API to watches, would you bring data from external API back into KRM APIs? I feel if there's an assumption like CRD will always overwrite the external API, not sure if there's value in reconciling sooner (vs eventually when the resource.generation++).

Depends on the use case. In ConfigSync we have multiple layers and user can decide which they want: 1. Periodic Remediation 2. (Reactive) Drift Remediation 3. Drift Prevention

It’s the old terraform (you’re the only actor) versus kubernetes resource model (reconciliation loops/annealing) conflict. In both cases they seem to have taken a conscious approach.

The problem with the first is that it implies a need for much more ordering (where TF benefits from having a resource graph, which has its own issues), as on a single run you’re expected to completely converge to the model.

The problem with the second is that you can’t be sure you can ever converge to the model, since it’s part of a he design to rerun rerun rerun until spec and status converge.

I'd expect that they're periodically querying the S3 API in this case. At a high-level, operators are supposed to continuously reconcile the current and desired states (which imply watching out for drift). Unfortunately, most external systems lack APIs to listen for changes, hence the polling.

If your controller is managing 20,000 buckets, you'll make 20k external API calls during controller startup full sync to verify buckets exist and their config is up-to-date (99.99% are) –during which you won't get to processing a new object soon enough, bc the workqueue is still full.

question: is there a reason a single controller needs to do this work vs partitioning for scale and blast radius containment?

Same in k8s with load-balancers. If you have 100 services with LBs, and someone modifies one behind the scenes, we don't know to reconcile, because (AFAICT) none of the clouds have watch APIs and they all have API quota. Polling 100 LBs every 10 seconds = 10 QPS of 99.999% do-nothing traffic.