Older entries:

• Multiple Robots.txt Files for Single Domain | April 1, 2009
• Live Search Begins Crawling JavaScript with MSNBot-Media | January 2, 2009
• Share Your Funny SEO Mistakes | December 18, 2008
• Stop GoogleBot From Indexing You At Busy Times | December 12, 2008
• Google's Video Crawler Not Respecting Robots.txt Directives? | December 1, 2008
• Ditched GoogleBot But Now Want To Make Friends Again? | November 14, 2008
• CSF (ConfigServer Security and Firewall) May Be Blocking GoogleBot | November 3, 2008
• Google Webmaster Tools Slower In Reporting Than Google Cache | September 24, 2008
• How To Tame GoogleBot | August 28, 2008
• Did Google Stop Crawling Blogger Blogs on Personalized Domains? | August 7, 2008
• Did Ask.com Stop or Slow Crawling the Web? | July 18, 2008
• Google Improves Flash Indexing Again | July 15, 2008
• Google Retraction: Blocking Regions Is Not Cloaking | July 3, 2008
• Updated: Google Says Blocking Countries Outside of the US is Against Policies | July 2, 2008
• Some Webmasters Are Banning Yahoo Slurp | June 30, 2008
• Can You Remove a Site From a Country Specific Google Search Engine? | June 20, 2008
• Can Google's First Click Free Program be Used for Web Search? | June 5, 2008
• Google Crawl Rate Drops: Google Responds? | May 2, 2008
• GoogleBot Getting Tired? Google's Spider Crawling Less Documents | May 1, 2008
• Google Now Crawling Content Behind Forms | April 14, 2008
• Is Google's AdWords Spider Lowercasing Destination URLs? | April 2, 2008
• Yahoo Slurp, Yahoo Search Crawler, Suffering From ADHD? | March 31, 2008
• MSNBot Again Failing Reverse DNS Test | March 11, 2008
• MSRBot Bot, Microsoft Bot Causing Havoc on Some Servers? | February 25, 2008
• Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler | February 13, 2008
• Can You Hide Text & Links From Your Users, If Search Engines Won't See? | January 22, 2008
• Google's Remove URLs Feature Removes For 90 Days Only | January 22, 2008
• Wordpress Installation Now Blocking Search Engines? | January 22, 2008
• Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP? | January 14, 2008
• Ask.com Fixes Crawler Issue With Badly-Formed URLs | January 4, 2008
• Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors? | December 31, 2007
• Microsoft Confirms Reverse DNS Bot Issue & Promises To Resolve | December 6, 2007
• MSNBot Reverse DNS Test Fails Requirements | December 5, 2007
• Live Search's Spam Fake Referrals Were Cloaking Tests, Says Microsoft | December 4, 2007
• Double Check Your Robots.txt: Google Testing New Crawler Directives | November 21, 2007
• Got ASP 2.0? GoogleBot Can Trip a Bug in the Code | November 8, 2007
• Google Says DNS Provider Afraid.org Blocks GoogleBot | October 16, 2007
• GoogleBot Not Sending IF_MODIFIED_SINCE Request? | October 9, 2007
• Googlebot DNS Caching Issue Reportedly Resolved | September 19, 2007
• Possible GoogleBot DNS Issues Causing Indexing Issues at Google.com | September 11, 2007
• Is a Robots.txt File Required for Search Engine Optimization? | August 31, 2007
• Managing the Robots.txt File for Sites Sharing Same Local Files | August 30, 2007
• Becoming a Googlebot for Testing Purposes | July 23, 2007
• Does Googlebot Index Faster When Integrating Google Custom Search? | July 16, 2007
• Yahoo! Supports Robots-Nocontent: Enabling Organic Search Page Section Targeting | May 3, 2007
• Verify The Bots Accessing Your Site: Is Google.com Sending That GoogleBot? | March 7, 2007
• Google & Search Engines Do Not Mind Bad HTML When Crawling | December 25, 2006
• MSNbot Adds Method For Verification | November 30, 2006
• Google Terminates AdWords Account for "Cookie Spidering"? | November 29, 2006
• Bot Attacks: Yes It Can Happen To You | November 8, 2006