Google: Using A CDN Or Cloud Hosting, Set Up Routine Tests To Ensure Googlebot Is Not Blocked

Dec 9, 2022 - 7:21 am 0 by

Google Cdns

Gary Illyes from Google loves posting his PSAs on Mastodon. His next PSA is that if you are using a CDN or cloud hosting service, then make sure to test up a routine test to ensure that the service is not blocking Googlebot or other search engine crawlers that you want to crawl and index your site.

Gary wrote on Mastodon "If you care about Googlebot/Bingbot/otherBot visiting your site and you're using solutions like CDN or f5, you should make a habit of going through every now and then the automatically generated firewall rules generated by some of these solutions. Sometimes they perceive crawler traffic as malicious and block it."

click for full size

Generally, this is good practice with any aspect of your website or software. You should set up routine tests that you can run manually on a schedule or just set to run in an automated way and be notified of the result. If Googlebot is blocked, you can jump on it before it becomes a larger issue. If your credit card form is broken, you can jump on it before losing out on many orders. Setting up automated or routine tests for sites or software is just good practice and maybe should be done to see if bots or people are being restricted on your website.

Forum discussion at Mastodon.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Bing Search

Happy 14th Birthday BingBot

Oct 3, 2024 - 7:11 am
Blog Administration

Programming Note: Rosh Hashanah 5785 - Happy & Healthy Jewish New Year

Oct 2, 2024 - 3:01 pm
Search Forum Recap

Daily Search Forum Recap: October 2, 2024

Oct 2, 2024 - 10:00 am
Bing Search

Bing Generative Search Experience Rolling Out

Oct 2, 2024 - 7:51 am
Google Search Engine Optimization

Google Warning On JavaScript-Generated Product Markup

Oct 2, 2024 - 7:41 am
Google Updates

October 2024 Google Webmaster Report

Oct 2, 2024 - 7:31 am
Previous Story: Google's John Mueller Says HTML Sitemaps Should Never Be Needed