Google: Disallowed URLs Through Robots.txt Does Not Affect Crawl Budget

Jun 21, 2019 - 6:40 am 0 by

Google Disallow

Gary Illyes from Google added to the big crawl budget article he published on the webmaster blog in January 2017 a new question and answer. It basically says that if you disallow a URL in your robots.txt file, then those URLs do not affect your crawl budget.

He added this question and answer:

Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?
A: No, disallowed URLs do not affect the crawl budget.

He posted last night that he added the Q&A for a former Googler, Pierre Far:

More details:

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Search Ranking Update, Recipe Blogs Drop, Google Hits Reviews & More SEO, PPC and Local - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: February 20, 2024

Feb 20, 2024 - 4:00 pm
Google Search Engine Optimization

Google: Sites Wrongfully Using The Google Indexing API Often Are Spammy & Low Quality

Feb 20, 2024 - 7:51 am
Google Search Engine Optimization

Google Merchant Center Updates Estimated Delivery Times Calculation

Feb 20, 2024 - 7:41 am
Google Search Engine Optimization

Google Merchant Center Requires Meta Labels On AI Generated Images

Feb 20, 2024 - 7:31 am
Google Maps

Google Local Pack Tests Restaurant Price Ranges

Feb 20, 2024 - 7:21 am
Google

Google Search Can Wait On Customer Service Hold For You

Feb 20, 2024 - 7:11 am
Previous Story: Summer Season Logos From Google, Bing, Yahoo & More