Older entries:

• DuckDuckGo Not Respecting Robots.txt Directives? | November 27, 2012
• Google: Robots.txt Files Must Be Smaller Than 500KB | January 30, 2012
• BingBot IP Ranges Updated | November 16, 2011
• Google: Remove The Robots.txt File Completely | January 6, 2011
• Google Published Detailed Specifications on Robots.txt | November 26, 2010
• Google: The Robots Meta Tag Is Cumulative (First Come First Serve) | July 30, 2010
• Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed | December 16, 2009
• MSNBot Crawl Delay Doesn't Delay | November 11, 2009
• Bing's MSNBot Crawl Happy? | October 23, 2009
• Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails | September 17, 2009
• DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b) | August 3, 2009
• MSNBot Clicking on Bing adCenter Search Ads? | July 29, 2009
• New MSNBot Named adidxbot Causing Trouble | July 7, 2009
• Tracking Search Bot Activity | April 6, 2009
• Multiple Robots.txt Files for Single Domain | April 1, 2009
• Live Search Begins Crawling JavaScript with MSNBot-Media | January 2, 2009
• Some Webmasters Are Banning Yahoo Slurp | June 30, 2008
• Can Google's First Click Free Program be Used for Web Search? | June 5, 2008
• Google Now Crawling Content Behind Forms | April 14, 2008
• Is Google's AdWords Spider Lowercasing Destination URLs? | April 2, 2008
• MSNBot Again Failing Reverse DNS Test | March 11, 2008
• Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler | February 13, 2008
• Can You Hide Text & Links From Your Users, If Search Engines Won't See? | January 22, 2008
• Google's Remove URLs Feature Removes For 90 Days Only | January 22, 2008
• Wordpress Installation Now Blocking Search Engines? | January 22, 2008
• Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP? | January 14, 2008
• Ask.com Fixes Crawler Issue With Badly-Formed URLs | January 4, 2008
• Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors? | December 31, 2007
• MSNBot Reverse DNS Test Fails Requirements | December 5, 2007
• Double Check Your Robots.txt: Google Testing New Crawler Directives | November 21, 2007
• Is a Robots.txt File Required for Search Engine Optimization? | August 31, 2007
• Managing the Robots.txt File for Sites Sharing Same Local Files | August 30, 2007
• Yahoo! Supports Robots-Nocontent: Enabling Organic Search Page Section Targeting | May 3, 2007
• Google & Search Engines Do Not Mind Bad HTML When Crawling | December 25, 2006
• Bot Attacks: Yes It Can Happen To You | November 8, 2006
• Yahoo! Slurping Wildcards Via Robots.txt File | November 6, 2006
• Scrape Bots Vs. Search Bots :: Fighting the Battle | September 12, 2006
• MSN Crawlers Are Named | August 1, 2006