Daniele Polencic

Daniele Polencic

@danielepolencic.com

Teaching Kubernetes at @LearnKube.com

292 followers11 following380 posts

Daniele Polencic reposted

"It's really expensive in people." Molly Sheets joins Daniele and Bart on the staffing cost of seven Kubernetes provisioning models https://ku.bz/Db3VnC3nl Sponsored by LearnKube

1/10 One of the things I like about the LearnKube Telegram group is the discussion that follows the articles we share. We shared a case study about a payment service stuck in a restart loop. Its liveness probe depended on a downstream service.

View full thread

9/10 Even after the downstream service recovers, the containers can still struggle because they compete for CPU. It is also why requests matter beyond placement. They influence how CPU is shared when the node is busy.

1/10 One of the things I like about the LearnKube Telegram group is the discussion that follows the articles we share. We shared a case study about a payment service stuck in a restart loop. Its liveness probe depended on a downstream service.

View full thread

7/10 The next question was: if the container needs all that CPU at startup, how did it work in the first place? Because it can use spare CPU beyond its request. A request is not a ceiling on CPU usage.

8/10 Starting one container while the others are quiet is different from restarting all ten together. The resource configuration is the same, but the available CPU is not.

1/10 One of the things I like about the LearnKube Telegram group is the discussion that follows the articles we share. We shared a case study about a payment service stuck in a restart loop. Its liveness probe depended on a downstream service.

View full thread

5/10 You still only have four CPUs, so the containers compete for them. With equal requests and no other load, each gets roughly 0.4 CPU. The startup work takes longer.

6/10 The code that answers the liveness probe needs CPU too. If the response takes too long, the probe fails. After enough consecutive failures, the kubelet restarts the container. And now you repeat the startup work, with the same competition for CPU.

1/10 One of the things I like about the LearnKube Telegram group is the discussion that follows the articles we share. We shared a case study about a payment service stuck in a restart loop. Its liveness probe depended on a downstream service.

View full thread

4/10 Imagine ten containers on a node with four CPUs. They use very little CPU during normal operation, but each wants a full CPU at startup. If they restart together, they all want that CPU at the same time.

1/10 One of the things I like about the LearnKube Telegram group is the discussion that follows the articles we share. We shared a case study about a payment service stuck in a restart loop. Its liveness probe depended on a downstream service.

2/10 The author said that repeated restarts caused CPU and memory pressure. Someone asked: can restarts really push containers into a resource-constrained state?

1/5 This KubeCon NA, we will host our first community event: LearnKube Day in Salt Lake City, on November 9. This one is special to me. For the first time, we will bring our community together for a day of our own. 🧵

View full thread

4/5 But the part I am most excited about is the chance to meet the people who supported us over the years. If you read our articles, took a course, listened to the podcast, or shared our work, thank you. I hope we can meet in person!

1/5 This KubeCon NA, we will host our first community event: LearnKube Day in Salt Lake City, on November 9. This one is special to me. For the first time, we will bring our community together for a day of our own. 🧵

2/5 To celebrate, Artem Lajko, Diana Todea, Viktor Farcic, Graziano Casto, Brandt Keller, Carol Valencia, and Jay Jackson will join us for lightning talks. Quite a group for our first event!

3/5 The day includes: • A hands-on observability and AI SRE workshop • Technical talks on Bank Grade Kubernetes, developer velocity, cost optimization, and AI operations • YAML Games, drinks, and time to talk

Agents are becoming part of everyday Kubernetes operations. How do we know when to trust them? Some mistakes produce an immediate error. Others look like successful fixes until production conditions change 🧵

View full thread

5/6 This is where I get stuck on evals Do platform teams need to maintain incident scenarios alongside their infrastructure to assess their agents? What evidence makes you trust a proposed fix when some mistakes only appear under load or during failover?

Agents are becoming part of everyday Kubernetes operations. How do we know when to trust them? Some mistakes produce an immediate error. Others look like successful fixes until production conditions change 🧵

View full thread

3/6 OpenAPI specifications, the Flux Schema MCP, versioned documentation, and live telemetry all help If an agent invents a CRD field, validation catches it. The agent reads the error and tries again But what if the answer looks correct?

4/6 Suppose the agent increases retries to fix intermittent failures The configuration is valid. A small test passes. At peak traffic, those retries amplify the overload There is no immediate feedback that the fix was wrong

Agents are becoming part of everyday Kubernetes operations. How do we know when to trust them? Some mistakes produce an immediate error. Others look like successful fixes until production conditions change 🧵

2/6 Two articles made me think about models that rely more on reasoning and tools, and less on memorized facts For platform ops, I like that direction. I want a generalist that retrieves accurate information about my system

View full thread

Saxo and Itau approached a related problem with operators Saxo reconciled external service dependencies, while Itau used a Kubernetes-style control plane to relocate services across runtimes The overlap is partial, but the contrast is striking

View full thread

Some decisions showed broad agreement. Others led to very different solutions The book covers tenant isolation, delivery, policy, multicluster networking, reliability, service dependencies, and cluster replacement

Financial firms have a tougher time with Kubernetes They face the usual problems around scaling, networking, and onboarding, plus regulatory compliance, auditing, and a massive workforce spread across on-premises infrastructure and multiple clouds

I wanted to understand how banks handle that complexity, so I partnered with @BuoyantIO, @sysdig, and @NirmataCloud and took a deep dive After a lot of research and the usual week spent illustrating my work, the result is a seven-chapter book

View full thread

Obviously, if downtime is not an option, this setup does not work for you. But 99.9% availability leaves about 8 hours and 45 minutes of downtime per year. For me, that is plenty for critical updates.

View full thread

But if you stick to the core resources and use YAML to describe your infrastructure, Kubernetes is surprisingly manageable. You can do a lot without getting into more complicated scenarios.

I know that I have replaced one controversial take with another: is a single node the antithesis of Kubernetes? Maybe. I prefer fewer moving parts because they make the system easier to reason about.

View full thread

ECS, EKS, or… EC2? If you had an expert whisperer next to you, would it really matter which orchestrator you chose? And are internal developer platforms becoming another harness?