Entries from Search Engine Roundtable tagged with 'user agent'

Colin Cochrane noticed that del.icio.us has blocked search engine spiders. He believes that it's not a simple robots.txt exclusion; instead, del.icio.us is serving 404 errors based on the User-Agent. Barry Welford confirmed this by changing the User-Agent himself. How did...

Posted on February 18, 2008 9:21 AM

Brendan Kowitz wrote a blog post about an interesting anomaly he noticed as Googlebot was crawling his site. Apparently, in March of 2006, Googlebot's User-Agent string was changed which triggered a bug in ASP.NET webservers. What this would mean is...

Posted on November 8, 2007 10:14 AM

Ever wanted to become a Googlebot to see what the bot sees? Now you can. While a Firefox Extension already does this, a DigitalPoint Forums thread discusses how to do this as well (for PC users). The idea is to...

Posted on July 23, 2007 9:37 AM