Most recent entries:
- Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site
| January 3, 2014
- Video Recap of Weekly Search Buzz :: February 1, 2013
| February 1, 2013
- DuckDuckGo Not Respecting Robots.txt Directives?
| November 27, 2012
- Google: Robots.txt Files Must Be Smaller Than 500KB
| January 30, 2012
- BingBot IP Ranges Updated
| November 16, 2011
- Blocking Google Video Bot Wont Stop Your Pages Showing In Google Video
| December 15, 2010
- Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed
| December 16, 2009
- MSNBot Crawl Delay Doesn't Delay
| November 11, 2009
- Bing's MSNBot Crawl Happy?
| October 23, 2009
- Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails
| September 17, 2009
- Stop Spiders From Crawling Your Site on Shabbat, Including GoogleBot
| September 9, 2009
- DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)
| August 3, 2009
- MSNBot Clicking on Bing adCenter Search Ads?
| July 29, 2009
- New MSNBot Named adidxbot Causing Trouble
| July 7, 2009
- Tracking Search Bot Activity
| April 6, 2009
- Multiple Robots.txt Files for Single Domain
| April 1, 2009
| January 2, 2009
- Did Google Stop Crawling Blogger Blogs on Personalized Domains?
| August 7, 2008
- Did Ask.com Stop or Slow Crawling the Web?
| July 18, 2008
- Google Improves Flash Indexing Again
| July 15, 2008
- Google Retraction: Blocking Regions Is Not Cloaking
| July 3, 2008
- Updated: Google Says Blocking Countries Outside of the US is Against Policies
| July 2, 2008
- Can You Remove a Site From a Country Specific Google Search Engine?
| June 20, 2008
- Can Google's First Click Free Program be Used for Web Search?
| June 5, 2008
- Google Now Crawling Content Behind Forms
| April 14, 2008
- Is Google's AdWords Spider Lowercasing Destination URLs?
| April 2, 2008
- Yahoo Slurp, Yahoo Search Crawler, Suffering From ADHD?
| March 31, 2008
- MSNBot Again Failing Reverse DNS Test
| March 11, 2008
- Google's Remove URLs Feature Removes For 90 Days Only
| January 22, 2008
- Wordpress Installation Now Blocking Search Engines?
| January 22, 2008
- MSNBot Reverse DNS Test Fails Requirements
| December 5, 2007
- Does Googlebot Index Faster When Integrating Google Custom Search?
| July 16, 2007
- Verify The Bots Accessing Your Site: Is Google.com Sending That GoogleBot?
| March 7, 2007
- Google Terminates AdWords Account for "Cookie Spidering"?
| November 29, 2006
- Scrape Bots Vs. Search Bots :: Fighting the Battle
| September 12, 2006