Post

Published my reading between the lines of OpenAI Kubernetes postmortem: ahmet.im/blog/openai-... Many lessons to learn here, IMO it comes down to: * unintentional coupling of data plane/control plane * not being able to shield k8s API from load * inherent risk of asynchronous service announcements

ahmet.imNotes on OpenAI Kubernetes outageLast week, OpenAI has suffered a several hours long outage and published a detailed postmortem about it. Highly recommend reading it. These technical reports are usually a gold mine for all large-scal...

4 reposts30 likes

„Mixing data and control plane“. That‘s an interesting topic and sounds clearer than it is. We kind of encourage reusing the compute control plane (kube) for app control plane purposes. It‘s so convenient. Where is the line when this fires back?

Nice notes. I keep coming back to api-machinery as one of the most critical (& interesting) technical SIGs. It must be possible to do efficient server-side filtering of read ops (a "where clause"). Like it or not, people treat the API as a database, and when it doesn't work that way, fireworks.