Post

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

Concrete example: imagine a Bucket resource creates/configures buckets on s3 API. Someone goes to the s3 console and edits bucket configuration. Unless your controller periodically queries s3 API, it won't overwrite the out-of-band modifications until next time Bucket resource is modified.

Ideally an operator would watch the resources it manages and revert drift as soon as possible, to minimize downtime. But in the case where watching is not possible, polling may be acceptable. For scaling, consider adding a component that centralizes the polling and transforms it into watches.

Depends on the use case. In ConfigSync we have multiple layers and user can decide which they want: 1. Periodic Remediation 2. (Reactive) Drift Remediation 3. Drift Prevention