Gary Illyes from Google added to the big crawl budget article he published on the webmaster blog in January 2017 a new question and answer. It basically says that if you disallow a URL in your robots.txt file, then those URLs do not affect your crawl budget.
He added this question and answer:
Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?
A: No, disallowed URLs do not affect the crawl budget.
He posted last night that he added the Q&A for a former Googler, Pierre Far:
Based on a question from @pierrefar , i just updated the Crawl Budget blog post to include the following:
— Gary "鯨理" Illyes (@methode) June 21, 2019
Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?
A: No, disallowed URLs do not affect the crawl budget.https://t.co/kVYj6XaNn0
More details:
Oh. Interesting. I understand that line the other way around, so if you disallow a useless directory, you will gain back crawl budget
— Gary "鯨理" Illyes (@methode) June 21, 2019
Forum discussion at Twitter.
