Darryl Ruggles

27 May

https://lckhd.eu/NKBNro I replaced my Hashnode/Vercel blog with a custom-built, full-featured serverless blogging platform on AWS, and the entire thing took under 10 hours of hands-on development.

https://lckhd.eu/d0s9yt #Kubernetes #GitOps #ArgoCD Keeping track of manual Kubernetes config changes done via Kubectl can work for a while but in the long run it gets quite challenging. Configuration drift creeps in, rollbacks get messy, and environments fall out of sync. The article below

discusses using a GitOps approach with ArgoCD where Git is the single source of truth. The steps included are to install ArgoCD, connect it to a GitHub repo, and define an Application that watches for changes. When you push a replica count update, the cluster reconciles on its own. It also covers

a real sync bug from a wrong repo path. Check this out from Pranav Bakare!

https://lckhd.eu/QTBTjN #Kubeflow #AmazonEKS #MLOps #Karpenter #LLM I've been using Kubeflow for many years now and find it a really useful package of AI/ML tools.

View full thread

The pipeline preps data, fine-tunes with LoRA, applies AWQ quantization and pushes the results to S3, all on Karpenter GPU nodes. Daniel Pepuho works through the problems he ran into along the way, from Karpenter removing GPU nodes too early to Python dependency conflicts and stale container

image references, which should help if you are getting started with training pipelines on EKS.

https://lckhd.eu/QTBTjN #Kubeflow #AmazonEKS #MLOps #Karpenter #LLM I've been using Kubeflow for many years now and find it a really useful package of AI/ML tools.

Kubeflow Pipelines lets you build workflows to process data, build images, train models, capture results and more, and it runs on any Kubernetes cluster. The example below runs Kubeflow Pipelines on an EKS cluster in AWS, next to an existing vLLM on Ray serving setup.

https://lckhd.eu/352jhD #Bedrock #AgentCore #Observability #Terraform #Agents Pretty much everyone is building agents now, but I suspect many aren't putting much thought into observability. A single agent might be using MCP servers, code sandboxes, a browser, and memory and when something goes

wrong you need to trace where it happened or which part burned through a huge number of tokens. Carlos Biagolini put together a LangGraph agent on AgentCore Runtime that uses all of these. The whole stack including the AgentCore resources are setup with terraform and everything flows into

CloudWatch GenAI Observability. There are other approaches, but this is an easy starting point to learn from.

https://lckhd.eu/5IxYyD #Kubernetes #ConfigMaps #GitOps #Kustomize #SRE ConfigMaps work really well in Kubernetes for storing config options. One key thing is how pods pick up updated values. Depending on how you read them, you may not see a new value for a while, or ever. Restarting the pod will

make new values show up, but that’s likely not what many people expect. Whether a pod sees a change depends on how the ConfigMap values are accessed. If you use environment variables or subPath mounts, the values won’t update. If you mount the ConfigMap as a directory, the files will update after

a delay (about a minute), but your app still has to notice and reload them. Check this out from Vivek Raj.

https://lckhd.eu/SL7sKg #EKS #Kubernetes #LLMOps #GitOps #PlatformEngineering Many teams are looking to host their own models and the Elastic Kubernetes Service (EKS) is a good choice for that.

You can take advantage of dynamic cloud resources and scale up or down when needed. The example below builds a small LLM platform using Terraform to set up the VPC and node groups, and ArgoCD for a GitOps approach. A FastAPI gateway handles requests,

Prometheus collects metrics, and secrets come from AWS Secrets Manager. The example model is Qwen2.5 1.5B running only on CPU with llama.cpp. A GPU node group is set up but scaled to zero until the rest of the platform is proven. Check this out from Emmanuel Akuffo!

https://lckhd.eu/ODyiRB #Helm #Kubernetes #DevOps #GitOps Dealing with resources in Kubernetes typically involves crafting YAML manifest files. A single app running in your cluster may need many objects, including a Deployment, Service, ConfigMap, Secret, Ingress, ServiceAccount, and a

HorizontalPodAutoscaler. With a number of apps this can all easily get multiplied. The article below is a primer on Helm for those new to it, covering how Chart.yaml, values.yaml, and templates combine to render plain Kubernetes manifests. Check this out from Parag Sezwar.

https://lckhd.eu/bnSMec #AWS #EKS #Kubernetes #WAF #Networking There are a lot of components involved in traffic flow in an Elastic Kubernetes Service (EKS) cluster on AWS. If users of an app start seeing errors, it won't be clear where the problem is. Components in the path include Route 53, the

View full thread

This article helps navigate all of these components and traces a request across multiple AZs, from DNS to a private RDS instance.

It covers IP target mode, and why setting up CloudFront does not block direct hits on a public ALB unless the ALB's security group is restricted to the CloudFront prefix list. Check this out from Himanshu Rahangdale.

Web Application Firewall (WAF), the ALB, the Kubernetes Service, the pod, and even a database.

https://lckhd.eu/tRSAKa #Kubernetes #Kueue #DynamicResourceAllocation #GPU #GKE Using Kubernetes is really all about sharing a set of resources across multiple workloads. This works well for CPU and memory but isn’t as straightforward for resources like GPUs. Even a small embedding job that only

needs 4 GB of GPU memory asks your cluster for 1 GPU and holds a full 24 GB card until the pod completes. Another team that needs just 4 GB of that same GPU has to wait.

This article walks through using Dynamic Resource Allocation (DRA) with Kueue. With it, quota shifts from a count of GPUs to the amount of GPU memory a job actually needs, read from MIG counters or claimed capacity. Check this out from Michał Żyliński.

https://lckhd.eu/VGkVCE #AWS #FinOps #Serverless #Terraform #CloudCostOptimization Most AWS accounts have some leftover resources that people created to test something or for a PoC and never used again. These can really add up over time in $$$s. The billing pages in your account tell you what you

are being charged for, but not much about what the resources were created for, if they are still being used, or if anything else depends on them.

The article describes a tool called Opti Advisr that shows idle EC2 and RDS instances based on CPU usage over time and can stop them for you. It uses a tag system to scope which resources are included. This is an interesting approach from Abhishek Kumar, Adithyatej and Dhruvi Bagga.

https://lckhd.eu/J7rLLO #Kubernetes #YAML #DevOps #PlatformEngineering Most of what people deal with in Kubernetes is YAML manifests. They can be confusing for people who are new to them, and getting the indentation right can drive many people nuts. It is something you learn by seeing examples

and, more importantly, by writing your own over time. The article below takes an example Pod and explains each of its fields, from apiVersion all the way to tolerations, securityContext and init containers. Arvind Verma’s article, combined with some hands-on time writing your own YAML files, is a

good way to get started.

lckhd.euObservability patterns for AI agents on Amazon EKS | Amazon Web ServicesTraditional monitoring shows an AI agent

https://lckhd.eu/8mW222 #OpenTelemetry #EKS #Agents #Observability #Bedrock Observability is often overlooked in many apps, and AI agents are no different. Tracing incoming requests through tool calls down to inferences against models in Bedrock, including costs, is important. The article below

runs two OpenTelemetry setups in parallel, with agents exporting data either directly to Langfuse or through a shared collector.

Strands Agents is used, and a header carries the context into the Bifrost gateway. The gateway adds span data with token counts and cost, and Langfuse shows traces from either approach. An environment variable switches between them. Check this out from Hari Muthusamy and Elamaran Shanmugam.

lckhd.euKarpenter vs Cluster Autoscaler on EKS: migrate with zero downtime and halve the billNode autoscaling on EKS has two serious options: the venerable Cluster Autoscaler (CA) and Karpenter....

https://lckhd.eu/mZarV4 #Karpenter #EKS #Kubernetes #AWS #FinOps Mostly idle worker nodes are very common in Kubernetes clusters. There are tools you can use to right-size these, including the Cluster Autoscaler (CAS) and Karpenter. CAS works with node groups and adjusts their sizes, and a new

node can take 2 to 4 minutes to be usable. Karpenter picks instance types directly from pod requirements and typically has a node ready in 40 to 90 seconds.

This article compares the two approaches and gives some great tips for migrating. Florient Dogbe covers Pod Disruption Budgets (PDBs), Spot handling, scheduling when changes can happen, and keeping the Karpenter controller off the nodes it manages.

https://lckhd.eu/BpfgdN #Terraform #GitHubActions #AmazonEKS #OIDC #IAC Using Terraform and CI/CD tools like GitHub Actions is a good approach to provisioning resources on AWS and EKS. Using short-lived tokens with OIDC rather than hardcoded credentials is always recommended. Setting the

View full thread

This example from Oleh Adam Dubnytskyy sets up GitHub Actions to deploy to AWS and EKS with a trust policy limited to main and pull requests, IAM permissions pinned to specific ARNs, and an EKS access entry allowing Terraform to create Kubernetes objects.

The example grants cluster admin, but the author is clear about tightening that, along with separate plan and apply roles. Check it out!

associated IAM role and trust relationship to follow the principle of least privilege is also important.

lckhd.euHow AI workloads are reshaping Kubernetes autoscalingKubernetes has quietly run much of the world’s containerized software on a division of labor between two control loops that rarely had to talk to each other. The scheduler places each pod on a node…

https://lckhd.eu/MgNuMo #Kubernetes #Autoscaling #GPU #DRA In some cases GPU pods can stay Pending in your Kubernetes cluster, and you may think no more GPUs are available. That can be true, but the autoscaler might also have added a node the scheduler won’t use, since the two loops don’t always

agree on what types of nodes are needed. The article below shows how Dynamic Resource Allocation (DRA) gives both loops a single way to describe topology and device sharing. The scheduler can now hold a group of pods as a single unit, while the autoscaler still looks pod by pod. Sajjan

Gundapuneedi and Neelendra Bhandari explain how Kueue provisioning requests bridge that gap for now, and what the SIG-Autoscaling group is designing to close it.

lckhd.euIntroducing Amazon ECS Gateway: A fully-managed ingress solution for ECS services | Amazon Web ServicesTry Amazon ECS Gateway, a new account-level resource that simplifies ingress networking for containerized workloads by dynamically provisioning and managing the necessary infrastructure on your behalf, with all resources created directly in your account.

https://lckhd.eu/NcPeCT #ECS #AWS #Containers #PlatformEngineering #LoadBalancing I'm a big fan of using the Elastic Container Service (ECS) on AWS for container-based deployments.

In most cases you'll need to set up ALBs to handle ingress. AWS just announced ECS Gateway, which is an account-level resource that doesn't provision anything until services connect.

ECS sets up the certificate, hostname, target group and a security group that only accepts ALB traffic. You still pay for the resources used, but the gateway itself doesn't have any extra cost. Looking forward to trying this out. Check out this intro from Channy Yun (윤석찬)!

lckhd.euStructured outputs for Anthropic Claude models on Amazon Bedrock | Amazon Web ServicesAmazon Bedrock now supports structured outputs for all Anthropic Claude models on both the bedrock-runtime and bedrock-mantle endpoints. Receive schema-compliant JSON that conforms to a defined structure, without custom parsing, retry logic, or post-processing, using JSON Schema output format and strict tool use.

https://lckhd.eu/zeyqDc #AmazonBedrock #Claude #JSONSchema #AgenticAI For many inference cases it is best to force the responses into a specific structure, but it isn't always possible to get the LLM to do this. Writing defensive code to strip markdown, parse the JSON, and handle the errors is

ugly. It usually works, but there should be a better way. Now there is. Amazon Bedrock can enforce your JSON schema during generation for Claude models on the runtime and Mantle endpoints.