Google Can Crawl Site Sections More Frequently & Determine Quality By Section

Sep 2, 2022 - 7:21 am 0 by

Steel Rust Google

Google's Gary Illyes said on the last Search Off the Record podcast that Google can crawl certain sections of your site more frequently and also infer the quality of certain sections of your site differently.

This came up at the 9:09 minute mark into the podcast but Glenn Gabe summarized it super well on Twitter. Glenn said "Google can infer from a site overall which areas they might need to crawl more frequently. E.g. if there's a blog subdirectory & there are signals that it's popular/important, then Google might want to crawl there more." " And it's not just update frequency, it's also about quality. E.g. if G sees a certain pattern is popular (folder), & people are talking about it & linking to it, then it's a signal that ppl like that directory," he added.

Here is the video embed:

Here is the transcript of this section on crawl frequency by section of the site:

Yeah. Because like we said, we don't have infinite space, so we want to index stuff that we think-- well, not we-- but our algorithms determine that it might be searched for at some point, and if we don't have signals, for example, yet, about a certain site or a certain URL or whatever, then how would we know that we need to crawl that for indexing?

And some things you can infer from-- for example, if you launch a new blog on your main site, for example, and you have a new /blog subdirectory, for example, then we can sort of infer, based on the whole site, whether we want to crawl a lot from that /blog or not.

Then here is the section on quality:

But it's not just update frequency. It's also the quality signals that the main site has.

So, for example, if we see that a certain pattern is very popular on the internet, like a slash product is very popular on the internet, and people on Reddit are talking about it, other sites are linking to URLs in that pattern, then it's a signal for us that people like the site in general.

While if you have something that people are not linking to, and then you are trying to launch a new directory, it's like, well, people don't like the site, then why would we crawl this new directory that you just launched?

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Volatility, Helpful Content Update Gone, Dangerous Search Results & Ads Confusion - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google Will Fight Site Reputation Abuse Spam Both With Manual Actions & Algorithms

Apr 16, 2024 - 7:51 am
Google

Google Search Tests Thumbs Up/Down Buttons In Product Grid Results

Apr 16, 2024 - 7:41 am
Google Ads

Google Ads "Similar Product" Carousel

Apr 16, 2024 - 7:31 am
Google Search Engine Optimization

Clarification: Google Search Supports Images Referenced From src Attribute

Apr 16, 2024 - 7:21 am
Google AdSense

Google AdSense New Ad Intents Formats - Links & Anchors In Content

Apr 16, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: April 15, 2024

Apr 15, 2024 - 4:00 pm
Previous Story: Dual Google People Also Ask Results In Different Languages