Google: We're Crawling Half Of The URLs Over HTTP/2

May 20, 2021 - 7:11 am 2 by

Google Network

Google's John Mueller said in his I/O talk that Google is now crawling "more than half of all URLs with HTTP/2." GoogleBot first started crawling over HTTP/2 for a limited number of URLs in November 2020 and now half-a-year later, over 50% of what GoogleBot crawls is done over HTTP/2.

John said this at the 4:16 mark into the video. He said "we’re excited to be able to share that we’re now crawling more than half of all URLs with HTTP/2."

How does this help you with SEO? Well, it doesn't help much out of maybe crawl budget issues, which is not an issue for the majority of the websites out there. John said "In SEO terms, this falls into the so-called crawl budget. Crawl budget is a mix of how many URLs Google wants to crawl from your website, the crawl demand, and how many your URLs our systems think your server can handle without problem, the crawl capacity. With HTTP/2 crawling, this gets a little bit more complicated because a single connection can include multiple URLs. Overall, we find that HTTP/2 crawling gives our systems the ability to request more URLs with a similar load on your servers. The decision to crawl with HTTP/2 is based on whether the server supports it and whether our systems determine there’s a possible efficiency gain. You don’t have to do anything special apart from enabling HTTPS and HTTP/2 support on your web server."

Here is the relevant slide:

Google Http2

John said "This means Googlebot, our crawler, won’t need to spend as much time crawling your server as before." John added "Now, since starting this experiment, we’re excited to be able to share that we’re now crawling more than half of all URLs with HTTP/2. Using stream multiplexing and header compression, the number of connections and the bandwidth have both gone down significantly. These improvements help both our crawling as well as your website serving infrastructure. Woohoo."

Here is the video:

Forum discussion at YouTube.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Report: 14,000+ Google Search Ranking Features Leaked

May 28, 2024 - 6:15 am
Search Forum Recap

Daily Search Forum Recap: May 27, 2024

May 27, 2024 - 10:00 am

In Face Of AI Overview Backlash, Google Updates Docs With How To Show Web Only Results & How To Give Feedback

May 27, 2024 - 7:51 am
Google Search Engine Optimization

Google's John Mueller Blasts The Concept Of Toxic Links, Again

May 27, 2024 - 7:41 am
Google Search Engine Optimization

Some Reporting Fewer Links Reported In Google Search Console

May 27, 2024 - 7:31 am

Google Images "See Exact Matches" Helps You Find Who Stole Your Images

May 27, 2024 - 7:21 am
Previous Story: JohnMu In Google I/O Adventure Game