Tag: robots

Google: Don't Re-block Pages After You Allowed Them To Crawl To See Noindex Tags

Mar 26, 2021

Google To Update Robots.txt & Robots Tags Help Docs - Asks For Feedback

Mar 18, 2021

Search Buzz Video Recap: Bing Spam Penalties, Google Local Ranking Update, Speed Reports & More

Nov 8, 2019

Google: Meta Robots Nofollow Is A Hint Also

Sep 12, 2019

Bing: We Never Supported Noindex In Robots.txt

Jul 3, 2019

Google Works To Make Robots Exclusion Protocol A Real Standard

Jul 1, 2019

Google On How To Block Your Staging Server

Aug 15, 2017

Google: Keep The Default CMS Robots.txt Setting

Dec 28, 2016

Unique: AppleBot Will Listen As If It Is GoogleBot

Dec 2, 2015

Google: Don't Make A Dynamically Generated robots.txt

Oct 29, 2015

Google: I'd Avoid Using NoIndex In Robots.txt File

Sep 9, 2015

Google Talks About The Robots.txt Directive For One Whole Hour

Aug 26, 2015

Video Recap of Weekly Search Buzz :: September 12, 2014

Sep 12, 2014

Google Revamps Webmaster Tools Robots.txt Tester Tool

Jul 17, 2014

Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site

Jan 3, 2014

Google: Keep Your Robots.txt Simple

Mar 29, 2013

DuckDuckGo Not Respecting Robots.txt Directives?

Nov 27, 2012

Google: Robots.txt Files Must Be Smaller Than 500KB

Jan 30, 2012

BingBot IP Ranges Updated

Nov 16, 2011

Google: Remove The Robots.txt File Completely

Jan 6, 2011

Google Published Detailed Specifications on Robots.txt

Nov 26, 2010

Google: The Robots Meta Tag Is Cumulative (First Come First Serve)

Jul 30, 2010

Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009

MSNBot Crawl Delay Doesn't Delay

Nov 11, 2009

Bing's MSNBot Crawl Happy?

Oct 23, 2009

Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails

Sep 17, 2009

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009