(2)
For news publishers included in Google News, you may have noticed a significant decline in your crawl stats within Google Webmaster Tools starting in March of this year.
If you did not, then that would be weird because Google started crawling Google News publishers more efficiently by supporting the if-modified-since.
Matt Cutts announced it on Twitter:
If you have a news site and it looks like Google is crawling less since ~3/7, don't worry. Google News added if-modified-since support then.
— Matt Cutts (@mattcutts) April 8, 2014So it's just Google News doing smarter crawling now. More info about if-modified-since: http://t.co/1TrtFVnypD
— Matt Cutts (@mattcutts) April 8, 2014I checked the crawl stats for this site and indeed, the crawl activity by GoogleBot has dropped but the index and ranking has not dropped:
Forum discussion at Twitter.
Comments:
Jacques Bouchard
04/09/2014 12:25 pm
As a user of Google. I'd really appreciate it if these improvements somehow made it easier to actually find up-to-date news and blog posts from the main search results. For me, that's Google's biggest failing in serving me what I'm looking for.
Anton
04/10/2014 12:23 pm
googlebot become smarter! It now crawl urls based on google white-lists.