Entries from Search Engine Roundtable tagged with 'crawl delay'

A Google Groups thread has a fairly simple but educational FAQ on how the "Set Crawl Rate" feature works in Google Webmaster Tools. In short, you can only set the crawl rate for a site on the domain or subdomain...

Posted on October 25, 2007 8:15 AM

Last week Tamar wrote about How to Stop Googlebot from Crawling Your Site Rapidly, so I thought I write about the opposite. How can you induce GoogleBot into crawling your site. Although there is no magic shot that guarantees inducement...

Posted on October 11, 2007 7:35 AM

A DigitalPoint Forums member is concerned about the heavy crawling performed by the Googlebot. He claims that Google is crawling 3GB of data from his site a month and wants to reduce this as it's a bit excessive in his...

Posted on September 27, 2007 8:57 AM