Google: Don't Dynamically Update Robots.txt File Multiple Times Per Day

Jan 20, 2025 - 7:11 am 0 by

Google Logo Dynamic

Google's John Mueller said that since the robots.txt file is cached by Google for about 24-hours, it does not make much sense to dynamically update your robots.txt file throughout the day to control controlling.

Google won't necessarily see that you don't want Google to crawl a page at 7am and then at 9am you do want Google to crawl that page.

John Mueller wrote on Bluesky in response to this post:

QUESTION:

One of our technicians asked if they could upload a robots.txt file in the morning to block Googlebot and another one in the afternoon to allow it to crawl, as the website is extensive and they thought it might overload the server. Do you think this would be a good practice?

(Obviously, the crawl rate of Googlebot adapts to how well the server responds, but I found it an interesting question to ask you) Thanks!

ANSWER:

It's a bad idea because robots.txt can be cached up to 24 hours ( developers.google.com/search/docs/... ). We don't recommend dynamically changing your robots.txt file like this over the course of a day. Use 503/429 when crawling is too much instead.

This is not new news, we covered this a decade ago under Google: Don't Make A Dynamically Generated robots.txt. We also knew the 24-hour thing back in 2010.

Forum discussion at Bluesky.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: February 11, 2025

Feb 11, 2025 - 10:00 am
Google Search Engine Optimization

Google: How To Tell If Google Sees Tabbed Navigation

Feb 11, 2025 - 7:51 am
Google Ads

Advertisers: Google LSA Lead Quality Drops After Auto Lead Credits

Feb 11, 2025 - 7:41 am
Bing Search

Bing Tests Dropping AI Labels From AI Answers

Feb 11, 2025 - 7:31 am
Google Ads

Google Ads Support Account Re-Verification During Ongoing Thread

Feb 11, 2025 - 7:21 am
Google Ads

Google Ads Gambling and Games Policy Update On April 14th

Feb 11, 2025 - 7:11 am
Previous Story: YouTube Gaming Controller Heart Sign