Google: We're Crawling Half Of The URLs Over HTTP/2

May 20, 2021 - 7:11 am 2 by

Google Network

Google's John Mueller said in his I/O talk that Google is now crawling "more than half of all URLs with HTTP/2." GoogleBot first started crawling over HTTP/2 for a limited number of URLs in November 2020 and now half-a-year later, over 50% of what GoogleBot crawls is done over HTTP/2.

John said this at the 4:16 mark into the video. He said "we’re excited to be able to share that we’re now crawling more than half of all URLs with HTTP/2."

How does this help you with SEO? Well, it doesn't help much out of maybe crawl budget issues, which is not an issue for the majority of the websites out there. John said "In SEO terms, this falls into the so-called crawl budget. Crawl budget is a mix of how many URLs Google wants to crawl from your website, the crawl demand, and how many your URLs our systems think your server can handle without problem, the crawl capacity. With HTTP/2 crawling, this gets a little bit more complicated because a single connection can include multiple URLs. Overall, we find that HTTP/2 crawling gives our systems the ability to request more URLs with a similar load on your servers. The decision to crawl with HTTP/2 is based on whether the server supports it and whether our systems determine there’s a possible efficiency gain. You don’t have to do anything special apart from enabling HTTPS and HTTP/2 support on your web server."

Here is the relevant slide:

Google Http2

John said "This means Googlebot, our crawler, won’t need to spend as much time crawling your server as before." John added "Now, since starting this experiment, we’re excited to be able to share that we’re now crawling more than half of all URLs with HTTP/2. Using stream multiplexing and header compression, the number of connections and the bandwidth have both gone down significantly. These improvements help both our crawling as well as your website serving infrastructure. Woohoo."

Here is the video:

Forum discussion at YouTube.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Weekend Volatility, Google On Search Leak, Elizabeth Tucker Interview & Apple Intelligence - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 14, 2024

Jun 14, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Weekend Volatility, Google On Search Leak, Elizabeth Tucker Interview, Apple Intelligence & More

Jun 14, 2024 - 8:01 am

Google Tests Multiple Featured Snippets Under From Sources Across The Web

Jun 14, 2024 - 7:51 am
Google Search Engine Optimization

Google: Sometimes Search Experiments Conflict Causing Issues

Jun 14, 2024 - 7:41 am
Google Maps

Google Business Profiles Websites No Longer Load - 404

Jun 14, 2024 - 7:31 am
Google Ads

Google Tests New Call & Location Extension Ad Format

Jun 14, 2024 - 7:21 am
Previous Story: JohnMu In Google I/O Adventure Game