Post

I wish Kubernetes did less. Mesos was a thin layer of resource management & platforms (aka frameworks) had all the opinions on how jobs should run The biggest downside of Mesos was Zookeeper just like the biggest downside of K8s is etcd. If that wasn't there k8s would scale and be easier to manage

0 reposts9 likes

Borg is widely adopted because of Google’s org chart and culture From everything I’ve heard it’s overly complicated (feature creep) but internal teams have been layering on top of layers to make specific things easier K8s tries to layer & be flexible (eg deployments) which doesn’t layer well

This is why you see alternative options in k8s. Things like Argo rollouts. Same primitives, different opinions K8s deployments are good enough for a lot of things but once you need something else it is more work than it should be

I wish k8s had a faster, in-memory db option like borg (I don’t think it uses that anymore) Borg scaled to 10k clusters by vertically scaling single masters at the cost of downtime for data replication during failover

If anyone runs nomad in production I’d love to hear your thoughts about what you like and don’t like about it It seems like a good balance between the complexity of k8s without giving up on all the flexibility

To be clear Borg uses (or used, last I touched it was ~10 years) a consensus algorithm and strongly consistent disk writes to checkpoint state. One of the reasons it scales is (or was) that it has a "pull" model - control-plane nodes poll worker nodes, so scale-out is the norm.

One aspect of k8s I am unhappy with is that we did not see an emergent ecosystem of specialized workload abstractions/controllers. It's just too technically tedious, still.

It's true that getting rid of things in Borg is nigh impossible - someone is using EVERY feature (even when that feature literally does nothing anymore :) Google has a high tolerance for "exposed wiring" sorts of UXes, which is not true in the wider world.

We can get k8s to 10s of thousands of nodes on straight etcd. The number of orgs who legitimately should use clusters of that scale is probably O(10s) globally. That said, one of the things that makes k8s powerful (IMO) is the consistent API semantics. It also makes it tempting to do bad things.