Tag: robots

Google: I'd Avoid Using NoIndex In Robots.txt File

Sep 9, 2015

Google Talks About The Robots.txt Directive For One Whole Hour

Aug 26, 2015

Video Recap of Weekly Search Buzz :: September 12, 2014

Sep 12, 2014

Google Revamps Webmaster Tools Robots.txt Tester Tool

Jul 17, 2014

Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site

Jan 3, 2014

Google: Keep Your Robots.txt Simple

Mar 29, 2013

DuckDuckGo Not Respecting Robots.txt Directives?

Nov 27, 2012

Google: Robots.txt Files Must Be Smaller Than 500KB

Jan 30, 2012

BingBot IP Ranges Updated

Nov 16, 2011

Google: Remove The Robots.txt File Completely

Jan 6, 2011

Google Published Detailed Specifications on Robots.txt

Nov 26, 2010

Google: The Robots Meta Tag Is Cumulative (First Come First Serve)

Jul 30, 2010

Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009

MSNBot Crawl Delay Doesn't Delay

Nov 11, 2009

Bing's MSNBot Crawl Happy?

Oct 23, 2009

Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails

Sep 17, 2009

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009

MSNBot Clicking on Bing adCenter Search Ads?

Jul 29, 2009

New MSNBot Named adidxbot Causing Trouble

Jul 7, 2009

Tracking Search Bot Activity

Apr 6, 2009

Multiple Robots.txt Files for Single Domain

Apr 1, 2009

Live Search Begins Crawling JavaScript with MSNBot-Media

Jan 2, 2009

Some Webmasters Are Banning Yahoo Slurp

Jun 30, 2008

Can Google's First Click Free Program be Used for Web Search?

Jun 5, 2008

Google Now Crawling Content Behind Forms

Apr 14, 2008

Is Google's AdWords Spider Lowercasing Destination URLs?

Apr 2, 2008

MSNBot Again Failing Reverse DNS Test

Mar 11, 2008