Post

Kubernetes isn't the only orchestrator that exists. Here's some features I wish it had youtu.be/9N9IOpyl3v8

I wish Kubernetes did less. Mesos was a thin layer of resource management & platforms (aka frameworks) had all the opinions on how jobs should run The biggest downside of Mesos was Zookeeper just like the biggest downside of K8s is etcd. If that wasn't there k8s would scale and be easier to manage

Borg is widely adopted because of Google’s org chart and culture From everything I’ve heard it’s overly complicated (feature creep) but internal teams have been layering on top of layers to make specific things easier K8s tries to layer & be flexible (eg deployments) which doesn’t layer well

This is why you see alternative options in k8s. Things like Argo rollouts. Same primitives, different opinions K8s deployments are good enough for a lot of things but once you need something else it is more work than it should be

I wish k8s had a faster, in-memory db option like borg (I don’t think it uses that anymore) Borg scaled to 10k clusters by vertically scaling single masters at the cost of downtime for data replication during failover

If anyone runs nomad in production I’d love to hear your thoughts about what you like and don’t like about it It seems like a good balance between the complexity of k8s without giving up on all the flexibility

Ha, catch up homie. I’ve been demoing multi cluster service discovery for four years. First using service messes then upstream MCS. IMO, MCG + MSC + MC WLID + Gitops gets you close to treating multiple clusters across multiple regions like one. We still need to MC Net pol and fee other things.

You can! (WAN Federation) Hashi licensing aside, nomad is one of the lowest maintenance systems I’ve ever deployed at scale (it’s almost zero touch at Fermyon, at CircleCI it was higher touch, but we were scheduling hundreds of jobs/min and that was before it had batch raft ops)

Nomad is great! But this is Nomad + Consul, right? Has anyone tried to sync/reconcile k8s service discovery from multiple clusters in different regions to a “global” consul? I’ve seen it done with the consul mesh or atnorth south boarder. Also @kelseyhightower.com making hashinetes back in the day.

I’ve only tried that with meshing - mostly bc routing would be annoying otherwise 😅 It works in nomad without consul (heck nomad even has its own service discovery now) - which I appreciate bc as powerful as consul can be, it has Many sharp edges 🙈