Post

Your ECR storage bill is not the problem. $0.10/GB-month, so 2 TB is ~$200/mo. The expensive part rides on stale images: Inspector rescanning all of them on every new CVE, replication copying every CI tag to every region. An agent that cleans it without deleting a thing 🧵

Contrarian bit first: the agent never calls batch-delete-image. Its only write path is a PR to the Terraform that owns the repos, adding a lifecycle policy. ECR does the expiring, on rules a human reviewed. An LLM with delete rights on a registry = Friday ImagePullBackOff.

The policy everyone copies from the docs (expire untagged after 7 days) fixes the SMALLEST bucket. 140-repo account: - untagged 61k images / 38% of size - never-pulled 48k / 41% - stale 9.4k / 15% - active 2.1k / 5% 84 of 140 repos had no policy. 12 had the untagged-only one.

Why never-pulled dominates: CI tags every commit SHA. Monorepo, 40 services, 30 merges/day = ~1,200 images/day, each a 400 MB tag pulled once by the job that built it, or never. No age rule touches them while they're under your threshold. You need a COUNT rule on the SHA pattern.

Collector is read-only boto3: DescribeRepositories, DescribeImages, GetLifecyclePolicy, DescribeRegistry, DescribePullThroughCacheRules. The field that makes it work: lastRecordedPullTime (per image since 2022, ~24h granularity). 90-day idle test absorbs the lag. Not a week.

Two numbers that lie. Summing imageSizeInBytes is not the bill: shared layers are stored once. Rank with it, bill from Cost Explorer TimedStorage-ByteHrs. Multi-arch: a tagged index points at platform manifests listed as UNTAGGED. 'Expire untagged' kills the arm64 half of a tag.

The in-use set is what makes this safe. Resolve EVERYTHING to digests, tags move: - running pods: containerStatuses[].imageID across every kube context - ReplicaSets retained by Deployments (revisionHistoryLimit default 10 = 10 rollback targets) - Helm history (history-max 10)

...plus Argo CD app history (Git revision, render manifests, extract images), Lambda container functions (Code.ResolvedImageUri is already a digest), every ACTIVE ECS task definition revision, digests named in Kyverno / signed-image allowlists. Union = protected, any age.

The LLM does ONE job: read 200 sample tags per repo and propose tagPatternList globs (sha-*, v*, main-*). Regex gets team tagging conventions wrong in surprising ways; a model doesn't, if it only fills a fixed schema. Count rule keeps max(revisionHistoryLimit, helm history) + 5.

Two ECR semantics the prompt must encode or the model gets them wrong: - an image with several tags matches if ANY tag matches a rule, so a 'keep' tag protects nothing. Size the count rule instead. - first matching rule wins by priority: untagged rule first, broad patterns last.

First run: preview failed on 3 of 140 repos. Same cause: a Deployment with revisionHistoryLimit: 50 set during an incident years ago, plus a Helm release in another namespace on an older tag. Nobody knew those were rollback targets. A hand-applied policy would have expired them.

Same PR also adds a pull-through cache rule for Docker Hub (base images fetched once per region, not once per node through the NAT gateway) and replication filters on the release prefix only. Gotcha: the Secrets Manager name must start with ecr-pullthroughcache/ or it's rejected.