Google has posted a new help document named Things to know about Google's web crawling. This document currently lists 9 things on how Google's web crawling works.
Google said this document was created "Based on questions we've received over the years, we've put together a resource page with basic educational information about crawling to better highlight various resources about crawling that are available to site owners."
Here is the short version of this document but check out the full document as well:
- What is crawling? In short, crawling is how Google "sees" the web
- We have many crawlers; they each have important jobs
- We perform repeat crawls to find the latest updates and to provide the freshest search results
- Frequent crawling is a good sign!
- Google's crawling has grown over time as pages have become more complex
- We optimize crawling automatically
- Google crawlers never go into paywall or subscription content without permission
- Site owners have control over what gets crawled, and how
- Our standard crawlers always respect websites' choices about how their content is accessed and used
This page is worth a read, read it over at Things to know about Google's web crawling.
Here is a screenshot of this page for archival purposes:
Forum discussion at X.
Image credit to Lizzi
