Entries from Search Engine Roundtable tagged with 'google spider'

A Google Groups thread has a detailed discussion around the topic of Google spider, GoogleBot, crawling too much. Sometimes servers can be overwhelmed by all the traffic it gets and automated crawlers, such as GoogleBot, can add a tremendous amount...

Posted on August 28, 2008 8:01 AM

There are threads at Google Groups and DigitalPoint Forums with multiple reports of Google not crawling Blogger hosted blogs, that are on custom or private domains (i.e. not on blogspot.com domains). Many have reported that the Googlebot crawling has stopped...

Posted on August 7, 2008 8:26 AM

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM