Entries from Search Engine Roundtable tagged with 'crawler'

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said: After...

Posted on November 20, 2009 8:52 AM

JohnMu from Google posted in a Google Webmaster Help thread that Google typically crawls a site's robots.txt file on a daily basis. This is the first time (at least that I can remember) I have seen a Googler make a...

Posted on September 3, 2009 8:42 AM

A Google Webmaster Help thread has a member upset that Google is crawling his site during times when his server is overloaded. Is there a way to tell GoogleBot to stay away during these times? JohnMu of Google said, yes...

Posted on December 12, 2008 8:25 AM

It seems like we have confirmed reports from a Googler in Google Groups that Google's video crawler, part of the GoogleBot family, is not playing nice. In short, even though you may be telling Google not to crawl your videos,...

Posted on December 1, 2008 8:14 AM

A WebmasterWorld thread has an advertiser complaining that Google's AdWords spider appears to be lowercasing the destination URLs they have. The thing is, the lowercase URLs for this webmaster don't work with the site and they don't have the time...

Posted on April 2, 2008 7:50 AM

The Live Search Blog announced several updated to their crawler. The first is a name change to reflect the upgrade, previously named msnbot/1.0, it is now named msnbot/1.1. The bulk of the changes include the HTTP Compression and Conditional Get...

Posted on February 13, 2008 7:28 AM

Earlier this week, we reported that Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors. In short, Ask.com's crawlers were crawling badly formed URLs, causing tons of 404 errors in web server log files. Vivek Pathak, Ask.com's Infrastructure Product...

Posted on January 4, 2008 7:03 AM

A WebmasterWorld thread is reporting several webmasters noticing that Ask.com's crawler has recently been generating tons of 404 (file not found) errors on their sites. The issue appears to stem from Ask.com auto inserting URL-Encoded spaces into the URL. URL-encoded...

Posted on December 31, 2007 6:53 AM