Tag: robots
Most recent entries:
- Google: Don't Re-block Pages After You Allowed Them To Crawl To See Noindex Tags
| March 26, 2021
- Google To Update Robots.txt & Robots Tags Help Docs - Asks For Feedback
| March 18, 2021
- Search Buzz Video Recap: Bing Spam Penalties, Google Local Ranking Update, Speed Reports & More
| November 8, 2019
- Google: Meta Robots Nofollow Is A Hint Also
| September 12, 2019
- Bing: We Never Supported Noindex In Robots.txt
| July 3, 2019
- Google Works To Make Robots Exclusion Protocol A Real Standard
| July 1, 2019
- Google On How To Block Your Staging Server
| August 15, 2017
- Google: Keep The Default CMS Robots.txt Setting
| December 28, 2016
Older entries:
- Unique: AppleBot Will Listen As If It Is GoogleBot
| December 2, 2015
- Google: Don't Make A Dynamically Generated robots.txt
| October 29, 2015
- Google: I'd Avoid Using NoIndex In Robots.txt File
| September 9, 2015
- Google Talks About The Robots.txt Directive For One Whole Hour
| August 26, 2015
- Video Recap of Weekly Search Buzz :: September 12, 2014
| September 12, 2014
- Google Revamps Webmaster Tools Robots.txt Tester Tool
| July 17, 2014
- Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site
| January 3, 2014
- Google: Keep Your Robots.txt Simple
| March 29, 2013
- DuckDuckGo Not Respecting Robots.txt Directives?
| November 27, 2012
- Google: Robots.txt Files Must Be Smaller Than 500KB
| January 30, 2012
- BingBot IP Ranges Updated
| November 16, 2011
- Google: Remove The Robots.txt File Completely
| January 6, 2011
- Google Published Detailed Specifications on Robots.txt
| November 26, 2010
- Google: The Robots Meta Tag Is Cumulative (First Come First Serve)
| July 30, 2010
- Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed
| December 16, 2009
- MSNBot Crawl Delay Doesn't Delay
| November 11, 2009
- Bing's MSNBot Crawl Happy?
| October 23, 2009
- Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails
| September 17, 2009
- DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)
| August 3, 2009
- MSNBot Clicking on Bing adCenter Search Ads?
| July 29, 2009
- New MSNBot Named adidxbot Causing Trouble
| July 7, 2009
- Tracking Search Bot Activity
| April 6, 2009
- Multiple Robots.txt Files for Single Domain
| April 1, 2009
- Live Search Begins Crawling JavaScript with MSNBot-Media
| January 2, 2009
- Some Webmasters Are Banning Yahoo Slurp
| June 30, 2008
- Can Google's First Click Free Program be Used for Web Search?
| June 5, 2008
- Google Now Crawling Content Behind Forms
| April 14, 2008
- Is Google's AdWords Spider Lowercasing Destination URLs?
| April 2, 2008
- MSNBot Again Failing Reverse DNS Test
| March 11, 2008
- Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler
| February 13, 2008
- Can You Hide Text & Links From Your Users, If Search Engines Won't See?
| January 22, 2008
- Google's Remove URLs Feature Removes For 90 Days Only
| January 22, 2008
- Wordpress Installation Now Blocking Search Engines?
| January 22, 2008
- Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP?
| January 14, 2008
- Ask.com Fixes Crawler Issue With Badly-Formed URLs
| January 4, 2008
- Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors?
| December 31, 2007
- MSNBot Reverse DNS Test Fails Requirements
| December 5, 2007
- Double Check Your Robots.txt: Google Testing New Crawler Directives
| November 21, 2007
- Is a Robots.txt File Required for Search Engine Optimization?
| August 31, 2007
- Managing the Robots.txt File for Sites Sharing Same Local Files
| August 30, 2007
- Yahoo! Supports Robots-Nocontent: Enabling Organic Search Page Section Targeting
| May 3, 2007
- Google & Search Engines Do Not Mind Bad HTML When Crawling
| December 25, 2006
