Entries from Search Engine Roundtable tagged with 'spider'

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said: After...

Posted on November 20, 2009 8:52 AM

A Google Webmaster Help thread has a member upset that Google is crawling his site during times when his server is overloaded. Is there a way to tell GoogleBot to stay away during these times? JohnMu of Google said, yes...

Posted on December 12, 2008 8:25 AM

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM