Google: How To Block GoogleBot Temporarily Vs Permanently

Apr 2, 2015 - 8:40 am 1 by

robot turtleThere is a short but informative post by Google's John Mueller in the Google Webmaster Help forum on the topic of blocking Google from seeing content on a specific page.

John explains there are many ways to block Google from indexing your page but the technique would be different based on the time period you want something blocked from indexing. Often I see that webmasters are confused about how to handle this, so here is John Mueller's advice:

  • If you just don't want the content indexed (maybe you're trying something out on the page), then using the robots.txt is a good approach
  • If it's very temporarily, maybe even a 503 HTTP response code
  • If you want the page actively removed from search, then I'd definitely recommend using a noindex over the robots.txt
  • If you're using a staging server and don't want that indexed, limiting access to just the testers' IP address ranges or using server-side authentication would be good approaches too.

John added that don't flip/flop back and forth between these techniques on a single page because it will confuse GoogleBot. He wrote:

One thing I'd try to avoid is quickly fluctuating back & forth. Removing content from search, and then bringing it back can sometimes result in us not recrawling as quickly as you'd like, and it therefore taking a bit longer for the pages to come back into search. If you can avoid that by running your tests (for example) on a separate site that's not indexed, that's generally preferred. Of course, if you're looking to remove something permanently, that's less of an issue.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 18, 2024

Mar 18, 2024 - 4:00 pm
Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Previous Story: Google AdSense Publishers Get Email On AdWords Social Casino Games Ads