Google: Firewalls or CDNs Issues Are The Most Common Reason For Blocking Googlebot

Feb 1, 2023 - 7:51 am 2 by

Google Fire Wall

Gary Illyes from Google posted a new PSA on LinkedIn saying that the most common reason a site unexpectedly blocks Googlebot from crawling is due to a misconfiguration of a firewall or CDN.

Gary wrote, "check what traffic your firewalls and CDN are blocking." "By far the most common issue in my inbox is related to firewalls or CDNs blocking googlebot traffic. If I reach out to the blocking site, in the vast majority of the cases the blockage is unintended."

So what can you do? Gary said, "I've said this before, but want to emphasize it again: make a habit of checking your block rules. We publish our IP ranges so it should be very easy to run an automation that checks the block rules against the googlebot subnets."

Gary linked to this help document for more details.

In short, do what you can to test to see if your site is accessible to Googlebot. You can use the URL inspection tool in Google Search Console, as one method. Also, confirm with your CDN or firewall company that they are allowing Googlebot and ask them to prove it.

Forum discussion at on LinkedIn.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google

Google Search AI Overviews Video Descriptions

Jul 23, 2024 - 7:51 am
Google Ads

Google Won't Deprecate Third-Party Cookies In Chrome

Jul 23, 2024 - 7:41 am
Google

Google Tests Larger Sitename & Favicons

Jul 23, 2024 - 7:31 am
Google Ads

New Google Ads Sitelinks Interface

Jul 23, 2024 - 7:21 am
Google Ads

Google Ads Editor Version 2.7 Is Now Out

Jul 23, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: July 22, 2024

Jul 22, 2024 - 10:00 am
Previous Story: Google: Search Console Verification Does Not Impact Your Ranking