How To Ask GoogleBot (Google) To Crawl Your Site

Microsoft adCenter to Add Daily Budget Feature | Main | Has Google Fixed the Proxy Hijack Problem? Google.com Cleaner

October 11, 2007

How To Ask GoogleBot (Google) To Crawl Your Site

Last week Tamar wrote about How to Stop Googlebot from Crawling Your Site Rapidly, so I thought I write about the opposite. How can you induce GoogleBot into crawling your site.

Although there is no magic shot that guarantees inducement of the crawl you can always increase your chances of that crawl in a few ways.

Submit a Sitemap file to Google Webmaster Tools
Make sure you have plenty of links pointing to your site from pages that are already in the Google index
Build out a real sitemap for users
Make sure you have a search engine friendly site
Increase the crawl rate at Google Webmaster Tools to "Faster"

I am sure there are many other ways to help induce a crawl, but here are just a few good ways to do it.

Forum discussion at DigitalPoint Forums.

Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Google Optimization at October 11, 2007 7:35 AM Comments (8)

Comments

re Increase the crawl rate at Google Webmaster Tools to "Faster"

As far as i can see this option is hardly ever available.

Posted by Peter at October 11, 2007 08:32

Any information regarding Googlebot crawl rate speed = faster would be of great help. I am new to web design, having started a New Media company I aim to add value by generating SEO friendly website design, researching the concept of a symantec web and contirbuting, where possible, to the development of structured on-line information

Posted by Rich09 at October 11, 2007 09:09

The option to set crawl rate to "faster" is available more often if you have old domain or sites which haven't been updated or doesn't contain any substantial content. You then launch a new site on this domain and sometimes you get a "faster" option but not always. It does work well if this option is available, it will usually hit your site for 3-4 months and then taper to a normal level.

Posted by Ben Pfeiffer at October 11, 2007 11:33

As mentioned in http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620 the "faster" setting for crawl rate is only available when it is determined that the Googlebot would be ready to crawl the site faster anyway. For sites that are not being crawled yet, this setting will likely not be available. Also, as mentioned, the setting will return to normal after 90 days. Keep in mind that "normal" could be faster or slower than before (it depends on many factors).

Posted by JohnMu at October 11, 2007 16:42

I turned my site to faster crawling but it is not indexing newer pages. For example, my last page added was on Feb. 11th..And it was added almost instantly. Since then it has not indexed any new pages..

Posted by Raptorak at February 26, 2009 00:59

Raptorak, just a thought, Do you have a sitemap.xml? If so have you tried to set the priority of your new page to 1. Does your last page added have a link form your homepage or the highest ranked page on your site?

Google in February has been very slow in indexing content. Yahoo Slurp is much faster at the moment.

Posted by Rob Abdul at February 26, 2009 09:05

it is quite useful information.i have set my site's crawl rate to faster. And i think its much better than normal because googlebot is indexing my pages faster than before.
Its nice and thanks to webmasterteam.

Posted by redown at October 15, 2009 11:23

Great information. But in google webmaster I still dont have the special setting for the crawl rate?

Posted by No Name at December 16, 2009 00:26

Premium Sponsors + advertise

All Out on the Table

Recent Table Activity

SEM Forum Threads

To subscribe to the Search Engine Roundtable, click here