Tag: crawler

New Retired Google Crawlers Document With DuplexWeb-Google Being Retired

Dec 5, 2022

GoogleBot Does Not Use Nutch

Oct 5, 2020

Bing: We Never Supported Noindex In Robots.txt

Jul 3, 2019

GoogleBot Favicon: Google Has A New Favicon Crawler

Oct 16, 2018

GoogleBot Grows Taller: Reaches 12,140 Pixels High On Mobile & 9,307 Pixels High On Desktop

Jan 18, 2018

GoogleBot Can Detect If Cookies Are Required

Mar 21, 2017

Google Says AdsBot Not Used For Web Search URL Discovery

Feb 7, 2017

How To Request A Special Crawl Rate From Google

Dec 30, 2016

Google Search Console Removed Feature Phone URL Error Report

Nov 30, 2016

Webmasters Claiming More AppleBot Activity From Apple

Feb 8, 2016

Video Recap of Weekly Search Buzz :: September 18, 2015

Sep 18, 2015

Google Webmaster Tools Updates Fetch & Render Report & Adds Blocked Resources Report

Mar 11, 2015

GoogleBot Crawling Too Hard & Much? Report The Issue To Google.

Feb 20, 2015

GoogleBot Skipping Unrendered Page & Crawling/Indexing Final Rendered Page

Sep 2, 2014

BingBot As GoogleBot? Maybe, If It Crawls Google's PageSpeed Tool

Aug 29, 2014

BingBot Crawl Activity Surging?

Jan 28, 2013

BingBot Hiding User Agent?

Dec 20, 2011

Bing's Spider, BingBot, Crawling Too Hard?

Jun 28, 2011

BingBot Arrives: Microsoft's New Crawler

Sep 29, 2010

GoogleBot Can Also Crawl Too Much & Be Nasty

Nov 20, 2009

Google Crawls Robots.txt Files Daily

Sep 3, 2009