Post

Hi @johnmu.com! I need your insights: I was reading the Crawl Budget documentation, and since it says crawl budget is calculated by host, I was wondering if resources coming from the host count in crawl budget? Do 3rd-party tools like bazaarvoice, for user reviews, count in their own crawl budget?

Yes, of course! My understanding is that whatever resources needed to properly render your page content will consume "crawl budget". Sometimes, these resources are not hosted on your domain, but on a commonized domain of a 3rd-party. Does the 3rd party domain also have its own crawl budget?

The most common example I think of is reviews on ecommerce websites, often loaded by a 3rd party solution like Bazaarvoice. I guess loading the JS snippet in my website "eats" my crawl budget, but I'm wondering if this tool's domain used to serve the snippet also has a crawl budget.

Got it - it's specific to the requested resource, it doesn't matter where it was embedded. The goal is to make sure that the number of requests sent to the server don't cause problems for the server.

1 repost2 likes