Post

What is the deal with everyone's bots. This is a brand new project (a few weeks ago) and they've literally crawled the whole thing and cost me > $100 in compute/memory in a few weeks. I want to allow searches but this feels like training data...

26/9/26, 4:12 pm0 reposts1 like

Peter Sugihara

@campsh.com

not excusing it but curious... why is it so expensive to serve requests?

John Nunemaker

@johnnunemaker.com

It’s a pretty basic site but pulls a lot of text out. On railway and they are usually quite cheap. But no requests are cached right now and unsure if it would have helped as all of them are just scanning new pages. And there are a lot of pages.

John Nunemaker

@johnnunemaker.com

I haven’t looked into that yet. I was just curious what was causing the traffic as it was brand new site I haven’t talked about.

Jakob Skjerning

@jakobs.bsky.social

It’s bonkers. One of our client sites has been serving 90% bot traffic the last few months 😥

John Nunemaker

@johnnunemaker.com

Yeah and it’s not real clear what is ok to block and what excludes you from the future.