Google: How To Block GoogleBot Temporarily Vs Permanently

Apr 2, 2015 - 8:40 am 1 by

robot turtleThere is a short but informative post by Google's John Mueller in the Google Webmaster Help forum on the topic of blocking Google from seeing content on a specific page.

John explains there are many ways to block Google from indexing your page but the technique would be different based on the time period you want something blocked from indexing. Often I see that webmasters are confused about how to handle this, so here is John Mueller's advice:

  • If you just don't want the content indexed (maybe you're trying something out on the page), then using the robots.txt is a good approach
  • If it's very temporarily, maybe even a 503 HTTP response code
  • If you want the page actively removed from search, then I'd definitely recommend using a noindex over the robots.txt
  • If you're using a staging server and don't want that indexed, limiting access to just the testers' IP address ranges or using server-side authentication would be good approaches too.

John added that don't flip/flop back and forth between these techniques on a single page because it will confuse GoogleBot. He wrote:

One thing I'd try to avoid is quickly fluctuating back & forth. Removing content from search, and then bringing it back can sometimes result in us not recrawling as quickly as you'd like, and it therefore taking a bit longer for the pages to come back into search. If you can avoid that by running your tests (for example) on a separate site that's not indexed, that's generally preferred. Of course, if you're looking to remove something permanently, that's less of an issue.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 29, 2025

Apr 29, 2025 - 10:00 am
Other Search Engines

ChatGPT Search Gains Shopping Search Features (Not Ads) & More

Apr 29, 2025 - 7:51 am
Google Search Engine Optimization

Google: Changing Lastmod Date In Sitemap Isn't An SEO Hack

Apr 29, 2025 - 7:41 am
Bing Search

Bing Tests New AI Answer Summary

Apr 29, 2025 - 7:31 am
Google Ads

Google Tests New Shopping Ads Design

Apr 29, 2025 - 7:21 am
Bing Search

Bing Search Without Microsoft Name By Logo

Apr 29, 2025 - 7:11 am
Previous Story: Google AdSense Publishers Get Email On AdWords Social Casino Games Ads