Post

You're correct. I'm in the not-normal category. Our GTM has a JS dependency to fire. We block the JS there. Good point on the Chrome not being a crawler...

With you here. If you have access to website logs that's always fun to see as well. What can I say. I have trust issues with Inspect?

Obsession with crawl budget. Hm. Seems like I found a new topic for my series "Questions I want to ask Google." :)

Yeah, easy to see these things as one size fits all, when really they aren't.

CSS & JS tend to be well-cachable (well, they should be), so this shouldn't be an issue for big sites either. (I know, I know, big sites can be very messy and all that, but the CSS isn't going to be their "only wafer thin mint")

Here's a possible example. 1. AI bots don't like javascript (someone only read a headline) 2. A game of telephone occurs. 3. If we block javascript on our robots.txt we're optimizing for AI bots! It's especially bad if the person giving the advice has a megaphone and an AI assistant references it.

That too! It was better known not to block these back when mobile-friendliness was a topic (you can't tell if it's mobile-friendly if you can't render it properly). It's one of those things that's obvious once you know, so it's good to post reminders from time to time :).

Ah, I see. I didn't know that

Agreed but in cases where JS & CSS are in security sensitive folders still block folders but ALLOW JS and CSS and test it because sometimes the order of robots commands can affect the ability for those files to be indexed. I've seen lots of cases WP admin folders exposed for JS and CSS indexation.

I don't know React that well - you can roughly test by blocking the URL in Chrome dev-tools and reloading the page. If the content loads & displays, it's probably fine.