Tag: robots

Google Published Detailed Specifications on Robots.txt

Nov 26, 2010

Google: The Robots Meta Tag Is Cumulative (First Come First Serve)

Jul 30, 2010

Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009

MSNBot Crawl Delay Doesn't Delay

Nov 11, 2009

Bing's MSNBot Crawl Happy?

Oct 23, 2009

Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails

Sep 17, 2009

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009

MSNBot Clicking on Bing adCenter Search Ads?

Jul 29, 2009

New MSNBot Named adidxbot Causing Trouble

Jul 7, 2009

Tracking Search Bot Activity

Apr 6, 2009

Multiple Robots.txt Files for Single Domain

Apr 1, 2009

Live Search Begins Crawling JavaScript with MSNBot-Media

Jan 2, 2009

Some Webmasters Are Banning Yahoo Slurp

Jun 30, 2008

Can Google's First Click Free Program be Used for Web Search?

Jun 5, 2008

Google Now Crawling Content Behind Forms

Apr 14, 2008

Is Google's AdWords Spider Lowercasing Destination URLs?

Apr 2, 2008

MSNBot Again Failing Reverse DNS Test

Mar 11, 2008

Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler

Feb 13, 2008

Can You Hide Text & Links From Your Users, If Search Engines Won't See?

Jan 22, 2008

Google's Remove URLs Feature Removes For 90 Days Only

Jan 22, 2008

Wordpress Installation Now Blocking Search Engines?

Jan 22, 2008

Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP?

Jan 14, 2008

Ask.com Fixes Crawler Issue With Badly-Formed URLs

Jan 4, 2008

Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors?

Dec 31, 2007

MSNBot Reverse DNS Test Fails Requirements

Dec 5, 2007

Double Check Your Robots.txt: Google Testing New Crawler Directives

Nov 21, 2007

Is a Robots.txt File Required for Search Engine Optimization?

Aug 31, 2007