Microsoft adCenter to Add Daily Budget Feature | Main | Has Google Fixed the Proxy Hijack Problem? Google.com Cleaner

October 11, 2007

How To Ask GoogleBot (Google) To Crawl Your Site

Last week Tamar wrote about How to Stop Googlebot from Crawling Your Site Rapidly, so I thought I write about the opposite. How can you induce GoogleBot into crawling your site.

Although there is no magic shot that guarantees inducement of the crawl you can always increase your chances of that crawl in a few ways.

  • Submit a Sitemap file to Google Webmaster Tools
  • Make sure you have plenty of links pointing to your site from pages that are already in the Google index
  • Build out a real sitemap for users
  • Make sure you have a search engine friendly site
  • Increase the crawl rate at Google Webmaster Tools to "Faster"

I am sure there are many other ways to help induce a crawl, but here are just a few good ways to do it.

Forum discussion at DigitalPoint Forums.



Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Google Optimization at October 11, 2007 7:35 AM Comments (4)

• Google Co-op Cartoon Barry

Comments

re Increase the crawl rate at Google Webmaster Tools to "Faster"

As far as i can see this option is hardly ever available.

 

Posted by Peter at October 11, 2007 08:32

Any information regarding Googlebot crawl rate speed = faster would be of great help. I am new to web design, having started a New Media company I aim to add value by generating SEO friendly website design, researching the concept of a symantec web and contirbuting, where possible, to the development of structured on-line information

 

Posted by Rich09 at October 11, 2007 09:09

The option to set crawl rate to "faster" is available more often if you have old domain or sites which haven't been updated or doesn't contain any substantial content. You then launch a new site on this domain and sometimes you get a "faster" option but not always. It does work well if this option is available, it will usually hit your site for 3-4 months and then taper to a normal level.

 

Posted by Ben Pfeiffer at October 11, 2007 11:33

As mentioned in http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620 the "faster" setting for crawl rate is only available when it is determined that the Googlebot would be ready to crawl the site faster anyway. For sites that are not being crawled yet, this setting will likely not be available. Also, as mentioned, the setting will return to normal after 90 days. Keep in mind that "normal" could be faster or slower than before (it depends on many factors).

 

Posted by JohnMu at October 11, 2007 16:42

Post a comment (Note: Can Take 120 Seconds For Your Comment To Show Up)

Do you want us to save your personal Information?

Premium Sponsors + advertise

To subscribe to the Search Engine Roundtable, click here