Post

Solid OpenAI post-mortem on @kubernetes.io API overload, caused by "per-node telemetry ingestion" status.openai.com/incidents/ct... 🤔 Oddly familiar to a common DaemonSet @prometheus.io / @opentelemetry.io gotcha for metrics scrapes, we talked about in the past: youtu.be/yk2aaAyxgKw?... 🙈

...there's even a OSS metric collection operator we maintain at Google with that optimization 🙃 github.com/GoogleCloudP... Another solution is to ofc have a cluster service discovery that pushes the targets to node agents, but pure (optimized) daemonset usually scales enough!

I had an idea in this area github.com/prometheus/p...