Post

Kubernetes isn't the only orchestrator that exists. Here's some features I wish it had youtu.be/9N9IOpyl3v8

I wish Kubernetes did less. Mesos was a thin layer of resource management & platforms (aka frameworks) had all the opinions on how jobs should run The biggest downside of Mesos was Zookeeper just like the biggest downside of K8s is etcd. If that wasn't there k8s would scale and be easier to manage

Borg is widely adopted because of Google’s org chart and culture From everything I’ve heard it’s overly complicated (feature creep) but internal teams have been layering on top of layers to make specific things easier K8s tries to layer & be flexible (eg deployments) which doesn’t layer well

This is why you see alternative options in k8s. Things like Argo rollouts. Same primitives, different opinions K8s deployments are good enough for a lot of things but once you need something else it is more work than it should be

I wish k8s had a faster, in-memory db option like borg (I don’t think it uses that anymore) Borg scaled to 10k clusters by vertically scaling single masters at the cost of downtime for data replication during failover

I inherited a nomad setup once. It went well (and by well I mean not) hazelweakly.me/blog/mother-...

I really like the two options of spreading clusters in multiple regions or running multiple clusters with workload discovery I’m trying to figure out the practical reasons to do one or the other

Twine is the most interesting orchestrator IMO because of their goal of 1M nodes and built primarily on top of systemd I found it hilarious that the whole paper they point out they don’t do federation like k8s and the very end of the paper that say they’re building federation for cross region tasks

Kubernetes had the benefit of being built with hindsight and the disadvantage of being applied universally

And also: Kubernetes doesn't do federation.

Blast radius. A single cluster presents a liability. Making it work across regions, even in the face of regional failure, means ALWAYS keeping that property as a dominant factor of every feature's design. K8s has multi-cluster services as a response to "why can't I just have multi-region clusters?"

I think MCS is only a GKE feature. Not K8s. IIRC this is because GKE uses GCP Cloud DNS and provides managed istio There are some open source projects to do similar things but none of them are something I would want to run in production

Ha, catch up homie. I’ve been demoing multi cluster service discovery for four years. First using service messes then upstream MCS. IMO, MCG + MSC + MC WLID + Gitops gets you close to treating multiple clusters across multiple regions like one. We still need to MC Net pol and fee other things.

I think nomad has something different. From what I can tell in the docs as an admin I can do something like k get pod and even if the pod is in a different cluster nomad will route me to the correct cluster to give me the information I could be wrong which is why I’m asking for someone w/ exp

You can! (WAN Federation) Hashi licensing aside, nomad is one of the lowest maintenance systems I’ve ever deployed at scale (it’s almost zero touch at Fermyon, at CircleCI it was higher touch, but we were scheduling hundreds of jobs/min and that was before it had batch raft ops)

It’s been a while but I think Calico also has a multi-cluster service discovery capability. (Not sure if it’s enterprise only though.)

We ran nomad for several years, now on EKS. I like to say that "k8s has an orchestrator, nomad IS an orchestrator." Like, that's all you get. Anything else you have to bolt on yourself. Less about "flexible" and more about "missing". It was an ok container stepping stone, but no ecosystem at all.

Sounds like ECS

I’m pretty sure Fly used but found they needed to build their own orchestrater over time. fly.io/blog/carving...