Here Is How Google May Use Machine Learning During Crawling

Nov 12, 2020 - 7:31 am 0 by

Google Machine Learning Crawling

Martin Splitt from Google said Google may be using machine learning in crawling in production right now, or it might be just an experiment, but here is how it does use machine learning for crawling in search. I should note, as far as he is aware, machine learning is not used for rendering in Google Search.

In yesterday's JavaScript SEO office hours with Martin Splitt of Google, I asked Martin at the 19:58 mark into the video if Google uses machine learning for crawling or rendering. Martin said he believes Google does for crawling but not for rendering.

How does Google uses machine learning for crawling? Two ways, for predicting quality of the URL before Google crawls it and for predicting the freshness of the URL before Google crawls it.

Predicting Quality

"I know we are using machine learning to identify or to predict what we will get from a crawl in terms of quality," Martin Splitt of Google said. He said it is interesting to try to "predict what kind of quality we can get from a specific crawl before it happens." He said the reason for that is because it would allow Google to "schedule crawling more intelligently."

Should Google bother using resources to crawl a URL that might not be worth its time, because the page is not high enough quality to warrant it is the question the ML is trying to answer.

Predicting Freshness

Martin Splitt said "the same goes for if we can predict freshness, as in do we know that we have to schedule this web site daily or is there, can we gather signals to tell us not to do this daily."

So Google can use machine learning to predict this for sites where Google does not have that much data for, i.e. content change frequency data.

He clarified "but I am not sure if we are using this in production or this is an experimental so far."

Here is the video embed at the start time:

Forum discussion at YouTube Community.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 19, 2024

Jul 19, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More

Jul 19, 2024 - 8:01 am
Google Search Engine Optimization

Billions Of Google URLs Will No Longer Work

Jul 19, 2024 - 7:51 am
Google Search Engine Optimization

Google: ccTLDs & Language Do Help You Rank A Little Better In Local Country Region

Jul 19, 2024 - 7:41 am
Google Search Engine Optimization

Google's On Knowing If Your SEO Team Is Doing Their Job

Jul 19, 2024 - 7:31 am
Google Ads

Google Merchant Center Next Gains Support For Supplemental Feeds

Jul 19, 2024 - 7:21 am
Previous Story: How Google Passage Indexing Might Look In The Search Results