Tag: spiders

The IRS Penalizes Bing?

Apr 28, 2014

Blocking Apple Devices On Your Web Site? You May Block Google Also!

Apr 16, 2014

Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site

Jan 3, 2014

Video Recap of Weekly Search Buzz :: February 1, 2013

Feb 1, 2013

DuckDuckGo Not Respecting Robots.txt Directives?

Nov 27, 2012

Google: Robots.txt Files Must Be Smaller Than 500KB

Jan 30, 2012

BingBot IP Ranges Updated

Nov 16, 2011

Blocking Google Video Bot Wont Stop Your Pages Showing In Google Video

Dec 15, 2010

Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009

MSNBot Crawl Delay Doesn't Delay

Nov 11, 2009

Bing's MSNBot Crawl Happy?

Oct 23, 2009

Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails

Sep 17, 2009

Stop Spiders From Crawling Your Site on Shabbat, Including GoogleBot

Sep 9, 2009

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009

MSNBot Clicking on Bing adCenter Search Ads?

Jul 29, 2009

New MSNBot Named adidxbot Causing Trouble

Jul 7, 2009

Tracking Search Bot Activity

Apr 6, 2009

Multiple Robots.txt Files for Single Domain

Apr 1, 2009

Live Search Begins Crawling JavaScript with MSNBot-Media

Jan 2, 2009

Did Google Stop Crawling Blogger Blogs on Personalized Domains?

Aug 7, 2008

Did Ask.com Stop or Slow Crawling the Web?

Jul 18, 2008

Google Improves Flash Indexing Again

Jul 15, 2008

Google Retraction: Blocking Regions Is Not Cloaking

Jul 3, 2008

Updated: Google Says Blocking Countries Outside of the US is Against Policies

Jul 2, 2008

Can You Remove a Site From a Country Specific Google Search Engine?

Jun 20, 2008

Can Google's First Click Free Program be Used for Web Search?

Jun 5, 2008

Google Now Crawling Content Behind Forms

Apr 14, 2008