Spiders
Older entries:
- Google Doesn't Use Toolbar Data To Index New URLs
| February 2, 2011
- Google: Remove The Robots.txt File Completely
| January 6, 2011
- Google: Canonical Tag Slower Than 301 Redirect
| December 8, 2010
- Structure Is King With SEO, Not Content
| December 6, 2010
- Google Published Detailed Specifications on Robots.txt
| November 26, 2010
- Google Fixes Blogspot Indexing Issue: New Content Being Indexed
| October 25, 2010
- Google Stop Indexing Blogger (Blogspot) Posts
| October 22, 2010
- Why Google May Not Find Your Sitemap File
| October 11, 2010
- Google Changes Coming Tomorrow: AJAX Results, 30 Results Per Page, Streaming Results & More?
| September 7, 2010
- Microsoft Cloaking MSNBots Again?
| September 3, 2010
- GoogleBot Crawling Less or More Efficiently?
| September 2, 2010
- GoogleBot Crawling From Different Locations At Same Time
| August 24, 2010
- Google: The Robots Meta Tag Is Cumulative (First Come First Serve)
| July 30, 2010
- Google Not Sending Referrer Data Again?
| July 13, 2010
- GoogleBot Acting Weird
| June 30, 2010
- BingBot Is Coming, BingBot Is Coming: Bye MSNBot
| June 30, 2010
- GoogleBot Back In Business, Crawling & Crawling
| June 25, 2010
- WordPress System Admin Calls Out Google's Evil Spiders
| June 23, 2010
- GoogleBot Stop Crawling Your Site On June 15th?
| June 16, 2010
- Bing Updates Language Detection After One or Two Crawls
| April 28, 2010
- Bing: We Know About Your Images & Choose Not To Show Them
| March 8, 2010
- Bing Masking MSNBot As Internet Explorer or a Rogue Bot?
| February 22, 2010
- Bing Indexing New Content & Sites Faster?
| February 17, 2010
- Google News Page Size Limit Is 256 KB
| January 13, 2010
- Microsoft Bing Says They Are "Fairly Slow"
| January 7, 2010
- Bing's MSNBot Crawling Fake File Names?
| December 28, 2009
- Google & Lightbox JavaScript: Can GoogleBot Index Images in Lightbox JS?
| December 28, 2009
- Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed
| December 16, 2009
- GoogleBot Can Also Crawl Too Much & Be Nasty
| November 20, 2009
- MSNBot Crawl Delay Doesn't Delay
| November 11, 2009
- Bing's MSNBot Crawl Happy?
| October 23, 2009
- Fetch As Googlebot Might Have Issues With Some Non-ASCII Characters
| October 21, 2009
- Fetch As Googlebot Shows Only 100Kb But Real Googlebot Gets More
| October 14, 2009
- Google Labs For Webmaster Tools Launches: Fetch as Googlebot & Malware Details
| October 13, 2009
- Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails
| September 17, 2009
- Stop Spiders From Crawling Your Site on Shabbat, Including GoogleBot
| September 9, 2009
- Bing Not Honoring Robots.txt Directives?
| September 9, 2009
- Google Crawls Robots.txt Files Daily
| September 3, 2009
- Microsoft Admits To Clicking On Their Own Search Ads
| August 13, 2009
- DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)
| August 3, 2009
- MSNBot Clicking on Bing adCenter Search Ads?
| July 29, 2009
- Microsoft's Search Bot MSNBot 2 Adding Pound Signs to URLs?
| July 20, 2009