Google: We're Crawling Half Of The URLs Over HTTP/2

May 20, 2021 - 7:11 am 2 by

Google Network

Google's John Mueller said in his I/O talk that Google is now crawling "more than half of all URLs with HTTP/2." GoogleBot first started crawling over HTTP/2 for a limited number of URLs in November 2020 and now half-a-year later, over 50% of what GoogleBot crawls is done over HTTP/2.

John said this at the 4:16 mark into the video. He said "we’re excited to be able to share that we’re now crawling more than half of all URLs with HTTP/2."

How does this help you with SEO? Well, it doesn't help much out of maybe crawl budget issues, which is not an issue for the majority of the websites out there. John said "In SEO terms, this falls into the so-called crawl budget. Crawl budget is a mix of how many URLs Google wants to crawl from your website, the crawl demand, and how many your URLs our systems think your server can handle without problem, the crawl capacity. With HTTP/2 crawling, this gets a little bit more complicated because a single connection can include multiple URLs. Overall, we find that HTTP/2 crawling gives our systems the ability to request more URLs with a similar load on your servers. The decision to crawl with HTTP/2 is based on whether the server supports it and whether our systems determine there’s a possible efficiency gain. You don’t have to do anything special apart from enabling HTTPS and HTTP/2 support on your web server."

Here is the relevant slide:

Google Http2

John said "This means Googlebot, our crawler, won’t need to spend as much time crawling your server as before." John added "Now, since starting this experiment, we’re excited to be able to share that we’re now crawling more than half of all URLs with HTTP/2. Using stream multiplexing and header compression, the number of connections and the bandwidth have both gone down significantly. These improvements help both our crawling as well as your website serving infrastructure. Woohoo."

Here is the video:

Forum discussion at YouTube.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Video Recaps

Search News Buzz Video Recap: Google Ranking Volatility Record, Forbes Advisor Slapped, Bing Generative Search Experience & More

Oct 4, 2024 - 8:01 am
Google

Google AI Overview Ads, New Link Format, AI Organized Search Results & More

Oct 4, 2024 - 7:55 am
Bing Search

Bing Testing Bing Button

Oct 4, 2024 - 7:51 am
Google Ads

Google Ads Send Message On WhatsApp Tests

Oct 4, 2024 - 7:41 am
Google Ads

Google Tests Local Services Ads Without Attorney Photos

Oct 4, 2024 - 7:31 am
Google

Google Search From The Web Carousel Cards

Oct 4, 2024 - 7:21 am
Previous Story: JohnMu In Google I/O Adventure Game