Ditched GoogleBot But Now Want To Make Friends Again?

Nov 14, 2008 - 7:07 am 0 by

I found an interesting tidbit while reading a somewhat detailed thread at Google Groups. The scenario is as follows. You have blocked Googlebot from accessing your site for a 6 month period or so. Then you want to welcome Googlebot back into your site by removing the disallow from your robots.txt file. Will Googlebot bite? If so, how long will it take?

Yea, you ditched Googlebot and now you realize what a good friend Googlebot can be for you. So now you want to become friends with little old Googlebot again. Truth be told, the longer you ditched Googlebot, possibly the longer Googlebot will accept your renewed friendship. Just like real friends. Well, not really.

It is just a matter of how many times Google will recheck your robots.txt file. If you blocked Googlebot in your robots.txt recently, then I would expect Googlebot to check the robots.txt more often. If it is six month or longer, than likely less frequently. It just makes sense from an efficiency standpoint on both Google's side and for your server.

Susan Moskwa of Google has a great analogy in the thread, so let me quote her:

I sometimes use a telephone metaphor to explain this--imagine you called someone and their answering machine said "I can't come to the phone right now, please don't call me." If you called back an hour later and got the same message, and then called back the next day and got the same message, and then called back the next week and got the same message, you'd probably call less and less frequently, assuming the response would be the same, right?

The same is true of Googlebot; if your site consistently sends the message "Don't crawl me," we may wait longer and longer between the times when we attempt to crawl it again.

So what can you do to encourage Googlebot to come back quicker? Send it flowers and chocolate. Just kidding! But you do want to entice Googlebot to come back. How so? Submit a Sitemap file to Google, get more links, get buzz out about your site, write more content and so on. As Susan said, "once it starts being 'active' in the online ecosystem, it'll naturally end up in search results and start ranking accordingly."

I had fun with this post because it is pretty late now, been up really long with little sleep. In fact, I feel like I am going to fall asleep while typing this. In any event, hope you enjoyed this little tidbit, it will be posted in the AM.

Forum discussion at Google Groups.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 22, 2024

May 22, 2024 - 10:00 am
Google Updates

Google Search Ranking Volatility Spike Around May 22

May 22, 2024 - 8:01 am
Google Ads

Google Ads In AI Overviews, AI Shopping Ads, Brand Profile Ads & More AI Ad Tools

May 22, 2024 - 7:51 am
Google Search Engine Optimization

Google & Bing Say Upgrade To WordPress 6.5 To Gain Lastmod Sitemap Support

May 22, 2024 - 7:41 am
Google Search Engine Optimization

Unusual Surge In Google Search Console Product Snippets Search Performance

May 22, 2024 - 7:31 am
Bing Search

Bing Search Testing Tag Filters

May 22, 2024 - 7:21 am
Previous Story: Google AdSense's Two Hour Reporting Outage