Google: How To Block GoogleBot Temporarily Vs Permanently

Apr 2, 2015 - 8:40 am 1 by

robot turtleThere is a short but informative post by Google's John Mueller in the Google Webmaster Help forum on the topic of blocking Google from seeing content on a specific page.

John explains there are many ways to block Google from indexing your page but the technique would be different based on the time period you want something blocked from indexing. Often I see that webmasters are confused about how to handle this, so here is John Mueller's advice:

  • If you just don't want the content indexed (maybe you're trying something out on the page), then using the robots.txt is a good approach
  • If it's very temporarily, maybe even a 503 HTTP response code
  • If you want the page actively removed from search, then I'd definitely recommend using a noindex over the robots.txt
  • If you're using a staging server and don't want that indexed, limiting access to just the testers' IP address ranges or using server-side authentication would be good approaches too.

John added that don't flip/flop back and forth between these techniques on a single page because it will confuse GoogleBot. He wrote:

One thing I'd try to avoid is quickly fluctuating back & forth. Removing content from search, and then bringing it back can sometimes result in us not recrawling as quickly as you'd like, and it therefore taking a bit longer for the pages to come back into search. If you can avoid that by running your tests (for example) on a separate site that's not indexed, that's generally preferred. Of course, if you're looking to remove something permanently, that's less of an issue.

Forum discussion at Google Webmaster Help.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 17, 2024

May 17, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More

May 17, 2024 - 8:01 am
Google Search Engine Optimization

Remove Your Content From Google's AI Overviews

May 17, 2024 - 7:51 am
Google Ads

Google Ads AI Summaries Live For Some Advertisers

May 17, 2024 - 7:41 am
Google Maps

Order with Google For Food Delivery Going Away End Of June

May 17, 2024 - 7:31 am
Google Search Engine Optimization

Two New Googlebots: GoogleOther-Image & GoogleOther-Video

May 17, 2024 - 7:21 am
Previous Story: Google AdSense Publishers Get Email On AdWords Social Casino Games Ads