For news publishers included in Google News, you may have noticed a significant decline in your crawl stats within Google Webmaster Tools starting in March of this year.
If you did not, then that would be weird because Google started crawling Google News publishers more efficiently by supporting the if-modified-since.
Matt Cutts announced it on Twitter:
If you have a news site and it looks like Google is crawling less since ~3/7, don't worry. Google News added if-modified-since support then.
— Matt Cutts (@mattcutts) April 8, 2014So it's just Google News doing smarter crawling now. More info about if-modified-since: http://t.co/1TrtFVnypD
— Matt Cutts (@mattcutts) April 8, 2014I checked the crawl stats for this site and indeed, the crawl activity by GoogleBot has dropped but the index and ranking has not dropped:
Forum discussion at Twitter.
