Entries from Search Engine Roundtable tagged with 'bots'

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM

We should have seen this coming, based on the number of reports that Google was submitting GET forms. But often, it is hard to validate those types of reports, due to people spoofing Googlebot and similar tactics. In any event,...

Posted on April 14, 2008 7:50 AM

A WebmasterWorld thread has an advertiser complaining that Google's AdWords spider appears to be lowercasing the destination URLs they have. The thing is, the lowercase URLs for this webmaster don't work with the site and they don't have the time...

Posted on April 2, 2008 7:50 AM

Last December, we reported that MSNBot was failing a reverse DNS lookup. Well, guess what folks - MSNBot is failing again on some IP addresses. An updated WebmasterWorld thread brought this to my attention and I verified it myself. Here...

Posted on March 11, 2008 6:59 AM

I noticed an update to the WebmasterWorld thread with the discussion of the weird referrals in the form of spam-like referrals coming from Live Search as cloaking tests. It appears a webmaster is now noticing a bot named MSLIVSOP serving...

Posted on January 14, 2008 7:27 AM

Yesterday we reported that MSNBot failed some reverse DNS tests. In short, if you did a reverse DNS lookup on the IP addresses of their spiders, it would not resolve to search.live.com. The big issue with this is that Webmasters...

Posted on December 6, 2007 7:20 AM

I have been corresponding with Microsoft about the weird spam-like referrals Live Search was sending to hundreds, if not thousands, of web site log files throughout the web. On September 6th, a Microsoft representative confirmed that these were actual tests...

Posted on December 4, 2007 3:00 PM

There is no doubt that a ton of bot activity on one's sites are from rogue spiders. Spider or bots that pretend to be legit bots but are there to steal your content. We have covered several sessions on this...

Posted on March 7, 2007 7:13 AM

We all know about PPC fraud and that some of the fraud is caused by bots (robots) that click on the ads and drive up your bill and unwanted traffic. But it gets more serious than that. Bot are also...

Posted on November 8, 2006 6:56 AM