Google On How To Block Your Staging Server

Aug 15, 2017 • 8:03 am | comments (5) by | Filed Under Google Search Engine Optimization

Google Stagging Crawl

Google's John Mueller posted on Google+ an overview of how to remove your staging server from Google and then the best way to block it going forward. Some call it a staging server, some a development server, some a testing server or some have all of these three and more. Heck, we even have pentest servers here, so we need to block them carefully from search engines and open them to those that need access.

First John explained that if these servers do find their content indexed by Google, the fastest way to remove the content is via Google Search Console. Of course that means you need to verify the staging URL with Google Search Console, but that is only long enough until you do the remove content method. After that, you then want to close the site from the outside world.

John wrote:

The fastest way to get the staging site removed from search is remove it via Search Console. For that, you need to verify ownership via Search Console [1] (ironically, this means you'll likely have to make it accessible to search engines again, or figure out DNS verification, which isn't that common but also not that hard). From there, you can do a site-removal request [2], which will take the whole hostname out of Google's search for ca 90 days. During this time, you can figure out and implement your general plan to block the staging site from search.

Then going forward, you want to lock down access to the server. John shared these tips, i.e. blocking on the server side with HTTP authentication or IP address whitelisting. He said all the other alternatives he does not like. John wrote:

My recommendation for staging sites is to block access on the server side, either with server-side / HTTP authentication [3] or IP address whitelisting (IP addresses can change, and this would block you from using tools from home, etc, so it's worth being cautious there and whitelisting rather than blacklisting).

He shared more details on Google+ adding that it's normal for URLs to not be recrawled in months, so if you add any block on the URL level, it can easily take a half year or longer to be fully processed for all URLs." "The site-removal request gives you most of that time, and you can submit another one should you need to extend it," he said.

Forum discussion at Google+.

Previous story: Google Search Console Structured Data Report Gets More Details
Ninja Banner
blog comments powered by Disqus