devtocash.bsky.social

devtocash.comBuild a Kubernetes PVC Right-Sizing Agent: Find Over-Provisioned Volumes with kubelet_volume_stats, Move gp2 to gp3, and Shrink the Ones Kubernetes Won't Let You ShrinkBuild a Kubernetes PVC right-sizing agent: find over-provisioned persistent volumes with kubelet_volume_stats, ship gp2-to-gp3 PRs, and migrate to shrink them.

Build a Kubernetes PVC right-sizing agent: find over-provisioned persistent volumes with kubelet_volume_stats, ship gp2-to-gp3 PRs, and migrate to shrink them. Full guide 👇 https://devtocash.com/blog/2026-10-27-kubernetes-pvc-right-sizing-agent-over-provisioned-volumes-gp2-to-gp3-shrink-migration

Your ops agent asked for a run_script tool. Fair: doing percentiles "in its head" is how you get confidently wrong postmortem math. Then the harness ran it via subprocess.run() inside the agent's own pod. The pod with the LLM key, MCP tokens, a SA token, metadata reach. Fix 🧵

Two honest limits. The /tmp fill is caught by kubelet eviction on an interval, so the write overshoots briefly. Fork bombs aren't in the pod spec: set podPidsLimit on the sandbox node pool. And stdout is STILL untrusted. A sandbox contains execution, not the text it returns.

devtocash.comAI Agent Sandbox on Kubernetes: Run LLM-Written Code in gVisor Jobs with No Network and No CredentialsBuild an AI agent sandbox on Kubernetes: run LLM-written scripts in ephemeral gVisor Jobs with no credentials, default-deny egress, hard timeouts, and quotas.

Full YAML, the ~60-line Python executor, and the 9-row escape-attempt matrix: https://devtocash.com/blog/2026-09-19-ai-agent-sandbox-kubernetes-gvisor-code-execution #Kubernetes #DevSecOps #DevOps

Executor ServiceAccount (held by the harness, never the model): a Role in agent-sandbox only. create+get jobs, list pods, get pods/log. No delete: the TTL controller cleans up, keeping finished Jobs inspectable 5 min and the quota rate-limit honest. Output capped at 16 KB.

Don't trust the YAML. Run hostile scripts through the tool in CI: - read SA token path = FileNotFoundError - urlopen 169.254.169.254 = timeout - gethostbyname = fails - while True = exit 124 - bytearray balloon = OOMKilled - 1 GB into /tmp = Evicted - no runtimeClassName = 403

Make the shape mandatory. A refactor that drops runtimeClassName silently downgrades every run to a plain container. ValidatingAdmissionPolicy (GA in 1.30) on the namespace: - runtimeClassName == 'gvisor' - no SA token automount - emptyDir volumes only PSA covers the rest.

The image is boring and pinned by digest: Python, jq, coreutils. No curl, no kubectl, no cloud CLIs. No network means no pip install at runtime, so bake numpy + pyyaml in. The tool description says NO network, NO creds, embed data inline. The model stops burning turns trying.

The script travels as a -c argument, not a ConfigMap. No extra object to clean up, no extra RBAC verb, and the exact code that ran is preserved in the Job spec for the audit trail. Linux caps a single argument at ~128 KiB, so the executor rejects scripts over 64 KB.

Layer 4: default-deny NetworkPolicy. podSelector {}, policyTypes Ingress + Egress, zero egress rules. No metadata endpoint, no internal services, no DNS. Resist "just allow kube-dns": a resolver is a data channel. On a CNI that ignores NetworkPolicy it protects nothing.

Layer 3: the Job template, non-obvious bits: - backoffLimit: 0 (never retry model-written code; the model reads the traceback) - automountServiceAccountToken: false - enableServiceLinks: false - runAsUser 65534, read-only rootfs, drop ALL caps - emptyDir /tmp 64Mi

Two timeouts, on purpose. Inner: timeout -k 2 60 python3 -c <script>. Exits 124 first, pod ends normally, partial output survives. Outer: activeDeadlineSeconds. When it fires the Job controller deletes the pod and its logs vanish. The backstop; also covers image pull.

Layer 2: gVisor RuntimeClass. runsc intercepts syscalls in a user-space kernel; the script never touches the node's kernel. GKE: node pool --sandbox type=gvisor. EKS/self-managed: register runsc in containerd. Pin it to the tainted pool or pods stick in FailedCreatePodSandBox.

Prove you're really sandboxed: run dmesg in a test pod. Under gVisor it prints gVisor's own boot messages instead of the node's kernel log. Can't run gVisor? Kata Containers behind the same RuntimeClass gives a microVM boundary; the rest of the design is unchanged.

Layer 1: an EMPTY namespace. Anyone who can create pods in a namespace can mount any Secret and use any ServiceAccount in it. So agent-sandbox holds nothing: no Secrets, no privileged SAs, no other workloads. Label it pod-security enforce=restricted.

A ResourceQuota that doubles as a rate limit: pods: 5 (max concurrent runs) count/jobs.batch: 30 With ttlSecondsAfterFinished: 300, a looping agent burns 30 executions, then gets 403 from the API server until old Jobs expire. Backpressure from the platform, not a prompt.

The script is untrusted input. The model needn't be malicious: it reads logs, PR comments and alert annotations, and any of those can carry an injection that ends in "now run this snippet". Deny it 4 things: credentials, network, the host kernel, unbounded time and resources.

Your Kubernetes PVCs are probably 3x too big. 400 PVCs at 100Gi requested / 15Gi used = roughly $3,000/mo of empty EBS blocks, and nobody looks because the pods are fine. 7 things a PVC right-sizing agent does that kubectl get pvc never will 🧵

Honest limits: filesystem usage is not data value (Postgres with no vacuum, a TSDB about to double retention). A 7-day predict_linear misses month-end step changes. Prices are us-east-1 list. The money is in the gp2 column anyway. Start there.

Full build: PromQL scan, classifier, LLM schema, gp3 patch, migration runbook, RBAC 👇 https://devtocash.com/blog/2026-10-27-kubernetes-pvc-right-sizing-agent-over-provisioned-volumes-gp2-to-gp3-shrink-migration #Kubernetes #FinOps #DevOps

StatefulSet trap: volumeClaimTemplates is immutable. Delete the StatefulSet with --cascade=orphan (pods + PVCs stay), recreate with the smaller template, swap claims by name. Wrong order = the controller creates fresh empty volumes. Data loss, not just FailedMount.

Rollout: one CronJob, a ServiceAccount with get/list on PVCs, PVs and storageclasses only. Two weeks of Markdown report in the FinOps channel before PR mode, and PR mode only for gp2 to gp3. In 3 clusters those PRs alone beat the orphaned-PVC sweep and merged in a day.

7. Shrinking: Kubernetes says no. spec.resources.requests.storage: Forbidden: field can not be less than previous value Snapshots don't help; a restore must be at least the snapshot's size. Only path: new smaller PVC + copy. So the agent writes a runbook, not an action.

The runbook: - scale writers to 0 (RWO can't mount twice) - apply the 60Gi gp3 claim - pv-migrate migrate old new --strategies mnt2,svc (no --ignore-mounted: the job refuses if step 1 was skipped) - du -sh /old /new before anything is deleted

5. The LLM writes the explanation, never the decision. Forced tool call; the schema has no field for a size, a type or a command. Only summary, risk_flags, workload_kind_guess. PVC names = untrusted data. '$38.88/mo' convinces nobody. 'Kafka at 2%, 500Gi x 6 brokers' does.

6. gp2 to gp3 is a PR with zero downtime. The EBS CSI driver modifies in place from PVC annotations: volumeType: gp3 iops: 3000 throughput: 125 On K8s 1.34+ use a VolumeAttributesClass instead. Rule the agent encodes: one EBS modification per volume per 6 hours.

3. Guards with predict_linear(kubelet_volume_stats_used_bytes[7d], 30d). A 100Gi volume at 12% used but projected 60Gi in 30 days is a log dir without rotation, not a shrink candidate. That 'growing 4x' finding is worth more than any savings number.

4. Prices honestly. gp2 $0.10/GiB-mo, gp3 $0.08. If it proposes gp2 to gp3 AND a shrink on the same volume, it prices the shrink at the gp3 rate. Double-count once and the first finance reviewer who checks the math stops trusting the whole list.

Why volumes end up 3x too big, on every cluster: - gp2 sized for IOPS: 3 IOPS/GiB, so 3,000 IOPS meant 1,000 GiB with 80 used - one volumeClaimTemplates 100Gi copy-pasted into Kafka, Redis and a 400 MB job queue - 'we can grow it later' was true, so the default became big

2. Classifies in code, not in the model. Named constants on the PR: MAX_USED_RATIO 0.30 HEADROOM 2.0 FLOOR 20 GiB MIN_SAVINGS $5/mo target = max(20, projected_30d x 2). Flag only if <30% used AND target < 60% of capacity.

1. Reads two numbers kubelet already exports for every mounted volume: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes then joins kube_persistentvolumeclaim_info with group_left. kubelet knows usage; only kube-state-metrics knows the storageclass and PV.

devtocash.comBuild a GitHub Actions Security Agent: Pin Actions to SHAs, Catch pull_request_target Pwn Requests, and Shrink GITHUB_TOKEN Permissions via PRBuild a GitHub Actions security agent: run zizmor org-wide, pin actions to full SHAs, block pull_request_target pwn requests, and ship least-privilege PRs.

Build a GitHub Actions security agent: run zizmor org-wide, pin actions to full SHAs, block pull_request_target pwn requests, and ship least-privilege PRs. Full guide 👇 https://devtocash.com/blog/2026-10-26-github-actions-security-agent-zizmor-pin-sha-pull-request-target-permissions-pr

Your ECR storage bill is not the problem. $0.10/GB-month, so 2 TB is ~$200/mo. The expensive part rides on stale images: Inspector rescanning all of them on every new CVE, replication copying every CI tag to every region. An agent that cleans it without deleting a thing 🧵

Same PR also adds a pull-through cache rule for Docker Hub (base images fetched once per region, not once per node through the NAT gateway) and replication filters on the release prefix only. Gotcha: the Secrets Manager name must start with ecr-pullthroughcache/ or it's rejected.

devtocash.comBuild an ECR Cost Agent: Expire Untagged and Never-Pulled Images, Generate Lifecycle Policies, and Prove No Cluster Still Runs What You DeleteBuild an ECR cost agent: find untagged and never-pulled images, prove nothing runs them, generate lifecycle policies, preview expirations, and ship them as PRs.

Full build: collector, in-use set from every consumer, classifier, the prompt, preview diff, and the limits list for the PR template: https://devtocash.com/blog/2026-10-16-ecr-cost-agent-untagged-images-lifecycle-policies-images-no-cluster-runs #AWS #FinOps #Kubernetes #DevOps

The free guardrail: aws ecr start-lifecycle-policy-preview, poll to COMPLETE, get-lifecycle-policy-preview into would_expire.txt. Then: comm -12 would_expire.txt inuse_all.txt. Any output = repo fails to a human with digests + workloads. No 'raise the count and retry' loop.

First run: preview failed on 3 of 140 repos. Same cause: a Deployment with revisionHistoryLimit: 50 set during an incident years ago, plus a Helm release in another namespace on an older tag. Nobody knew those were rollback targets. A hand-applied policy would have expired them.

The LLM does ONE job: read 200 sample tags per repo and propose tagPatternList globs (sha-*, v*, main-*). Regex gets team tagging conventions wrong in surprising ways; a model doesn't, if it only fills a fixed schema. Count rule keeps max(revisionHistoryLimit, helm history) + 5.

Two ECR semantics the prompt must encode or the model gets them wrong: - an image with several tags matches if ANY tag matches a rule, so a 'keep' tag protects nothing. Size the count rule instead. - first matching rule wins by priority: untagged rule first, broad patterns last.

The in-use set is what makes this safe. Resolve EVERYTHING to digests, tags move: - running pods: containerStatuses[].imageID across every kube context - ReplicaSets retained by Deployments (revisionHistoryLimit default 10 = 10 rollback targets) - Helm history (history-max 10)

...plus Argo CD app history (Git revision, render manifests, extract images), Lambda container functions (Code.ResolvedImageUri is already a digest), every ACTIVE ECS task definition revision, digests named in Kyverno / signed-image allowlists. Union = protected, any age.

Collector is read-only boto3: DescribeRepositories, DescribeImages, GetLifecyclePolicy, DescribeRegistry, DescribePullThroughCacheRules. The field that makes it work: lastRecordedPullTime (per image since 2022, ~24h granularity). 90-day idle test absorbs the lag. Not a week.

Two numbers that lie. Summing imageSizeInBytes is not the bill: shared layers are stored once. Rank with it, bill from Cost Explorer TimedStorage-ByteHrs. Multi-arch: a tagged index points at platform manifests listed as UNTAGGED. 'Expire untagged' kills the arm64 half of a tag.

The policy everyone copies from the docs (expire untagged after 7 days) fixes the SMALLEST bucket. 140-repo account: - untagged 61k images / 38% of size - never-pulled 48k / 41% - stale 9.4k / 15% - active 2.1k / 5% 84 of 140 repos had no policy. 12 had the untagged-only one.

Why never-pulled dominates: CI tags every commit SHA. Monorepo, 40 services, 30 merges/day = ~1,200 images/day, each a 400 MB tag pulled once by the job that built it, or never. No age rule touches them while they're under your threshold. You need a COUNT rule on the SHA pattern.