Tag: robots
Most recent entries:
- Microsoft Working On Content Controls For Bing Search vs Bing Chat
| April 3, 2023
- ChatGPT Bot - You Can Block OpenAI Plugins If You Want
| March 29, 2023
- Google Search Console Adds If embedURL Page Uses indexifembedded
| March 23, 2023
- Google: URLs Excluded By Robots.txt Aren't Removed Until URLs Are Individually Reprocessed
| March 22, 2023
- Microsoft: No Site Blacklist For Bing Chat Sidebar
| March 20, 2023
- Google Will Ignore Robots.txt Rules If It Serves A 4xx Status Code
| January 17, 2023
- Search News Buzz Video Recap: Google Still Rolling Out Updates, Start & End Dates Are Confusing, Bing To Add ChatGPT, SEO Is Bad & More
| January 6, 2023
- 500/503 HTTP Status Code On Robots.txt Can Remove Your Site From Google Search
| December 14, 2022
Older entries:
- Spammers Ruined Bing's Anonymous Sitemap Submissions, Microsoft Deprecated It
| May 16, 2022
- Search News Buzz Video Recap: Google Algorithm Fluctuations, FLoC FLoPs, New Google Robots Tag & More
| January 28, 2022
- New Google Robots Tag indexifembedded: Control Indexing With Embedded Content
| January 24, 2022
- Google: Don't Re-block Pages After You Allowed Them To Crawl To See Noindex Tags
| March 26, 2021
- Google To Update Robots.txt & Robots Tags Help Docs - Asks For Feedback
| March 18, 2021
- Search Buzz Video Recap: Bing Spam Penalties, Google Local Ranking Update, Speed Reports & More
| November 8, 2019
- Google: Meta Robots Nofollow Is A Hint Also
| September 12, 2019
- Bing: We Never Supported Noindex In Robots.txt
| July 3, 2019
- Google Works To Make Robots Exclusion Protocol A Real Standard
| July 1, 2019
- Google On How To Block Your Staging Server
| August 15, 2017
- Google: Keep The Default CMS Robots.txt Setting
| December 28, 2016
- Unique: AppleBot Will Listen As If It Is GoogleBot
| December 2, 2015
- Google: Don't Make A Dynamically Generated robots.txt
| October 29, 2015
- Google: I'd Avoid Using NoIndex In Robots.txt File
| September 9, 2015
- Google Talks About The Robots.txt Directive For One Whole Hour
| August 26, 2015
- Video Recap of Weekly Search Buzz :: September 12, 2014
| September 12, 2014
- Google Revamps Webmaster Tools Robots.txt Tester Tool
| July 17, 2014
- Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site
| January 3, 2014
- Google: Keep Your Robots.txt Simple
| March 29, 2013
- DuckDuckGo Not Respecting Robots.txt Directives?
| November 27, 2012
- Google: Robots.txt Files Must Be Smaller Than 500KB
| January 30, 2012
- BingBot IP Ranges Updated
| November 16, 2011
- Google: Remove The Robots.txt File Completely
| January 6, 2011
- Google Published Detailed Specifications on Robots.txt
| November 26, 2010
- Google: The Robots Meta Tag Is Cumulative (First Come First Serve)
| July 30, 2010
- Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed
| December 16, 2009
- MSNBot Crawl Delay Doesn't Delay
| November 11, 2009
- Bing's MSNBot Crawl Happy?
| October 23, 2009
- Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails
| September 17, 2009
- DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)
| August 3, 2009
- MSNBot Clicking on Bing adCenter Search Ads?
| July 29, 2009
- New MSNBot Named adidxbot Causing Trouble
| July 7, 2009
- Tracking Search Bot Activity
| April 6, 2009
- Multiple Robots.txt Files for Single Domain
| April 1, 2009
- Live Search Begins Crawling JavaScript with MSNBot-Media
| January 2, 2009
- Some Webmasters Are Banning Yahoo Slurp
| June 30, 2008
- Can Google's First Click Free Program be Used for Web Search?
| June 5, 2008
- Google Now Crawling Content Behind Forms
| April 14, 2008
- Is Google's AdWords Spider Lowercasing Destination URLs?
| April 2, 2008
- MSNBot Again Failing Reverse DNS Test
| March 11, 2008
- Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler
| February 13, 2008
- Can You Hide Text & Links From Your Users, If Search Engines Won't See?
| January 22, 2008
