Post

ahmet alp balkan
@ahmet.dev

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

ahmet alp balkan
@ahmet.dev

CR is the intended config, so I expect it to overwrite the reality –the question is whether it should try to course-correct periodically. There are some systems (like Prodspec/Annealing) that learn from the current state and bring it back to the configuration SoT, but they're few and rare.

0 reposts1 like

Ryanne Dolan
@ry.codes

My interpretation is that the CR is the intended state during reconciliation, not forever. If something/someone changes the actual state, they'll be surprised to find it asynchronously revert. I don't think the CR can realistically be a SoT most of the time.

ahmet alp balkan
@ahmet.dev

For most folks that ship has sailed. KRM-based tools like Crossplane manage infra in many shops. That’s why probably as a platform team, the moment you give users access to CRs, take the access away to modify the resource another way.

Ryanne Dolan
@ry.codes

Yeah that worries me. My team's operators own their resources -- even we can't manipulate them ourselves. But there is still drift. Some state transitions are irreversible, for example.

ahmet alp balkan
@ahmet.dev

Yeah we have a few APIs like that too, they don't play well with the "assume an open world" assumption of KRM. We do our best to validate the state transitions in webhooks. IMO not being able to manipulate them is a feature. :P when we need to manipulate we set a field called `spec.paused:true`.