Google: Using A CDN Or Cloud Hosting, Set Up Routine Tests To Ensure Googlebot Is Not Blocked

Dec 9, 2022 - 7:21 am 0 by

Google Cdns

Gary Illyes from Google loves posting his PSAs on Mastodon. His next PSA is that if you are using a CDN or cloud hosting service, then make sure to test up a routine test to ensure that the service is not blocking Googlebot or other search engine crawlers that you want to crawl and index your site.

Gary wrote on Mastodon "If you care about Googlebot/Bingbot/otherBot visiting your site and you're using solutions like CDN or f5, you should make a habit of going through every now and then the automatically generated firewall rules generated by some of these solutions. Sometimes they perceive crawler traffic as malicious and block it."

click for full size

Generally, this is good practice with any aspect of your website or software. You should set up routine tests that you can run manually on a schedule or just set to run in an automated way and be notified of the result. If Googlebot is blocked, you can jump on it before it becomes a larger issue. If your credit card form is broken, you can jump on it before losing out on many orders. Setting up automated or routine tests for sites or software is just good practice and maybe should be done to see if bots or people are being restricted on your website.

Forum discussion at Mastodon.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Flux, AdSense Ad Intent, California Link Tax & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Blog Administration

Programming Note: Offline For Passover

Apr 22, 2024 - 4:05 pm
Search Forum Recap

Daily Search Forum Recap: April 22, 2024

Apr 22, 2024 - 4:00 pm
Google Search Engine Optimization

Google: We Need Very Few Links To Rank Pages; Links Are Less Important

Apr 22, 2024 - 7:51 am

Google Tests Discussion and Forums Questions With Answers Box

Apr 22, 2024 - 7:41 am
Google Search Engine Optimization

Forbes Blocks Its Coupon Directory Before Google Enforces The Site Reputation Abuse Policy

Apr 22, 2024 - 7:31 am
Google Maps

Google Tests Posts Button In Local Listing Business Profile

Apr 22, 2024 - 7:21 am
Previous Story: Google's John Mueller Says HTML Sitemaps Should Never Be Needed