Google: How To Block GoogleBot Temporarily Vs Permanently

Apr 2, 2015 - 8:40 am 1 by

robot turtleThere is a short but informative post by Google's John Mueller in the Google Webmaster Help forum on the topic of blocking Google from seeing content on a specific page.

John explains there are many ways to block Google from indexing your page but the technique would be different based on the time period you want something blocked from indexing. Often I see that webmasters are confused about how to handle this, so here is John Mueller's advice:

  • If you just don't want the content indexed (maybe you're trying something out on the page), then using the robots.txt is a good approach
  • If it's very temporarily, maybe even a 503 HTTP response code
  • If you want the page actively removed from search, then I'd definitely recommend using a noindex over the robots.txt
  • If you're using a staging server and don't want that indexed, limiting access to just the testers' IP address ranges or using server-side authentication would be good approaches too.

John added that don't flip/flop back and forth between these techniques on a single page because it will confuse GoogleBot. He wrote:

One thing I'd try to avoid is quickly fluctuating back & forth. Removing content from search, and then bringing it back can sometimes result in us not recrawling as quickly as you'd like, and it therefore taking a bit longer for the pages to come back into search. If you can avoid that by running your tests (for example) on a separate site that's not indexed, that's generally preferred. Of course, if you're looking to remove something permanently, that's less of an issue.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 28, 2024

Mar 28, 2024 - 4:00 pm
Google Ads

Google Ads Suspended 90% More Advertisers This Year & Removed 5.5 Billion Ads

Mar 28, 2024 - 7:51 am
Google Ads

Google Updates Its Definition Of Top Ads; They May Not Be At The Top

Mar 28, 2024 - 7:41 am
Google Ads

Google Ads Adds Share Ad Preview For Performance Max

Mar 28, 2024 - 7:31 am
Google

Google Can Search For Your Blockchain Wallet Addresses (Bitcoin & More)

Mar 28, 2024 - 7:21 am
Google Maps

New Google Shopping & Maps Search Features

Mar 28, 2024 - 7:11 am
Previous Story: Google AdSense Publishers Get Email On AdWords Social Casino Games Ads