Post

Man - AI has completely eaten up the clouds capacity. We straight up can’t do a Postgres major version upgrade because GCP doesn’t have the capacity available for a measly 64cpu/256gb mem in their “Enterprise Plus” node pool. Scheduled 2 maintenance windows and failed twice now. Ridiculous!

Wow - apparently Google left us in a broken state where our read replicas are no longer able to replicate. Apparently the only solution there is to blow away our read replicas and re-create them. Except now that we’ve done that - GCP still doesn’t have any resources.

So now we can’t recreate our read replicas. Which don’t even need much resources in the first place! 32cpu and 128mem in 2 regions is all we’re asking for. We even have those resources back! But now they’ve already been allocated because everything is so oversubscribed.

The future is so dumb. We can’t even operate normal software anymore because chatbot throughput is more important.

Ok now THIS is crazy. I’m hearing from our GCP rep that the reason we can’t provision any new database instances is because gcp-west1 has completely exhausted its SSD capacity