Google Can Crawl Site Sections More Frequently & Determine Quality By Section

Sep 2, 2022 - 7:21 am 0 by

Steel Rust Google

Google's Gary Illyes said on the last Search Off the Record podcast that Google can crawl certain sections of your site more frequently and also infer the quality of certain sections of your site differently.

This came up at the 9:09 minute mark into the podcast but Glenn Gabe summarized it super well on Twitter. Glenn said "Google can infer from a site overall which areas they might need to crawl more frequently. E.g. if there's a blog subdirectory & there are signals that it's popular/important, then Google might want to crawl there more." " And it's not just update frequency, it's also about quality. E.g. if G sees a certain pattern is popular (folder), & people are talking about it & linking to it, then it's a signal that ppl like that directory," he added.

Here is the video embed:

Here is the transcript of this section on crawl frequency by section of the site:

Yeah. Because like we said, we don't have infinite space, so we want to index stuff that we think-- well, not we-- but our algorithms determine that it might be searched for at some point, and if we don't have signals, for example, yet, about a certain site or a certain URL or whatever, then how would we know that we need to crawl that for indexing?

And some things you can infer from-- for example, if you launch a new blog on your main site, for example, and you have a new /blog subdirectory, for example, then we can sort of infer, based on the whole site, whether we want to crawl a lot from that /blog or not.

Then here is the section on quality:

But it's not just update frequency. It's also the quality signals that the main site has.

So, for example, if we see that a certain pattern is very popular on the internet, like a slash product is very popular on the internet, and people on Reddit are talking about it, other sites are linking to URLs in that pattern, then it's a signal for us that people like the site in general.

While if you have something that people are not linking to, and then you are trying to launch a new directory, it's like, well, people don't like the site, then why would we crawl this new directory that you just launched?

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 15, 2024

Mar 15, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More

Mar 15, 2024 - 8:01 am
Google Updates

Google March 2024 Core & Spam Update Movement Today

Mar 15, 2024 - 7:51 am
Google

Google Image Search Results Testing Like Button

Mar 15, 2024 - 7:41 am
Google

Google Merchant Center Product Studio With Themed Templates Including St. Patrick's Day

Mar 15, 2024 - 7:31 am
Google Maps

Google Search Dishes Nearby Carousel

Mar 15, 2024 - 7:21 am
Previous Story: Dual Google People Also Ask Results In Different Languages