Entries from Search Engine Roundtable tagged with 'bots'

Back in the day, tracking how bots accessed your site was a bit of a crave. Now, you don't hear about it much. The old Google Analytics, aka Urchin, had a section for displaying bot activity on your site. It...

Posted on April 6, 2009 10:08 AM

Yesterday, we reported that Google's John Mueller said that if you block a whole region from accessing your site, it would be considered cloaking and thus be against Google's Webmaster guidelines. Since then, we have seen many comments on that...

Posted on July 3, 2008 1:24 PM

A Google Groups thread has a webmaster who has been receiving a lot of rogue spider attacks from the Africa region. He wants to go as far as ban the whole continent of Africa. But he is concerned that by...

Posted on July 2, 2008 7:40 AM

Last night, I had a nice chat with Googler, JohnMu. I joked around with John, asking if he has messed up yet, in terms of Google communication with webmasters. He said not really - which I agree with. But he...

Posted on June 5, 2008 10:05 AM

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM

We should have seen this coming, based on the number of reports that Google was submitting GET forms. But often, it is hard to validate those types of reports, due to people spoofing Googlebot and similar tactics. In any event,...

Posted on April 14, 2008 7:50 AM

A WebmasterWorld thread has an advertiser complaining that Google's AdWords spider appears to be lowercasing the destination URLs they have. The thing is, the lowercase URLs for this webmaster don't work with the site and they don't have the time...

Posted on April 2, 2008 7:50 AM

Last December, we reported that MSNBot was failing a reverse DNS lookup. Well, guess what folks - MSNBot is failing again on some IP addresses. An updated WebmasterWorld thread brought this to my attention and I verified it myself. Here...

Posted on March 11, 2008 6:59 AM

I noticed an update to the WebmasterWorld thread with the discussion of the weird referrals in the form of spam-like referrals coming from Live Search as cloaking tests. It appears a webmaster is now noticing a bot named MSLIVSOP serving...

Posted on January 14, 2008 7:27 AM

Yesterday we reported that MSNBot failed some reverse DNS tests. In short, if you did a reverse DNS lookup on the IP addresses of their spiders, it would not resolve to search.live.com. The big issue with this is that Webmasters...

Posted on December 6, 2007 7:20 AM

I have been corresponding with Microsoft about the weird spam-like referrals Live Search was sending to hundreds, if not thousands, of web site log files throughout the web. On September 6th, a Microsoft representative confirmed that these were actual tests...

Posted on December 4, 2007 3:00 PM

There is no doubt that a ton of bot activity on one's sites are from rogue spiders. Spider or bots that pretend to be legit bots but are there to steal your content. We have covered several sessions on this...

Posted on March 7, 2007 7:13 AM

We all know about PPC fraud and that some of the fraud is caused by bots (robots) that click on the ads and drive up your bill and unwanted traffic. But it gets more serious than that. Bot are also...

Posted on November 8, 2006 6:56 AM