Google: Using A CDN Or Cloud Hosting, Set Up Routine Tests To Ensure Googlebot Is Not Blocked

Dec 9, 2022 - 7:21 am 0 by

Google Cdns

Gary Illyes from Google loves posting his PSAs on Mastodon. His next PSA is that if you are using a CDN or cloud hosting service, then make sure to test up a routine test to ensure that the service is not blocking Googlebot or other search engine crawlers that you want to crawl and index your site.

Gary wrote on Mastodon "If you care about Googlebot/Bingbot/otherBot visiting your site and you're using solutions like CDN or f5, you should make a habit of going through every now and then the automatically generated firewall rules generated by some of these solutions. Sometimes they perceive crawler traffic as malicious and block it."

click for full size

Generally, this is good practice with any aspect of your website or software. You should set up routine tests that you can run manually on a schedule or just set to run in an automated way and be notified of the result. If Googlebot is blocked, you can jump on it before it becomes a larger issue. If your credit card form is broken, you can jump on it before losing out on many orders. Setting up automated or routine tests for sites or software is just good practice and maybe should be done to see if bots or people are being restricted on your website.

Forum discussion at Mastodon.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 27, 2024

Mar 27, 2024 - 4:00 pm
Google Ads

Google Local Service Ads Sends Email Asking You To Upload Photos

Mar 27, 2024 - 7:51 am
Google Maps

European Hotels Can Remove Pricing Information From Google Hotel Search

Mar 27, 2024 - 7:41 am
Google Search Engine Optimization

SEOs, Please Don't Remove Contact Us & About Us Pages

Mar 27, 2024 - 7:31 am
Misc Google

Google Search Developer Docs Gain AI Generated Help Features

Mar 27, 2024 - 7:21 am
Google Maps

Google Local Reviews Reactions Notice

Mar 27, 2024 - 7:11 am
Previous Story: Google's John Mueller Says HTML Sitemaps Should Never Be Needed