Google On How To Block Your Staging Server

Aug 15, 2017 - 8:03 am 5 by

Google Stagging Crawl

Google's John Mueller posted on Google+ an overview of how to remove your staging server from Google and then the best way to block it going forward. Some call it a staging server, some a development server, some a testing server or some have all of these three and more. Heck, we even have pentest servers here, so we need to block them carefully from search engines and open them to those that need access.

First John explained that if these servers do find their content indexed by Google, the fastest way to remove the content is via Google Search Console. Of course that means you need to verify the staging URL with Google Search Console, but that is only long enough until you do the remove content method. After that, you then want to close the site from the outside world.

John wrote:

The fastest way to get the staging site removed from search is remove it via Search Console. For that, you need to verify ownership via Search Console [1] (ironically, this means you'll likely have to make it accessible to search engines again, or figure out DNS verification, which isn't that common but also not that hard). From there, you can do a site-removal request [2], which will take the whole hostname out of Google's search for ca 90 days. During this time, you can figure out and implement your general plan to block the staging site from search.

Then going forward, you want to lock down access to the server. John shared these tips, i.e. blocking on the server side with HTTP authentication or IP address whitelisting. He said all the other alternatives he does not like. John wrote:

My recommendation for staging sites is to block access on the server side, either with server-side / HTTP authentication [3] or IP address whitelisting (IP addresses can change, and this would block you from using tools from home, etc, so it's worth being cautious there and whitelisting rather than blacklisting).

He shared more details on Google+ adding that it's normal for URLs to not be recrawled in months, so if you add any block on the URL level, it can easily take a half year or longer to be fully processed for all URLs." "The site-removal request gives you most of that time, and you can submit another one should you need to extend it," he said.

Forum discussion at Google+.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 22, 2024

May 22, 2024 - 10:00 am
Google Updates

Google Search Ranking Volatility Spike Around May 22

May 22, 2024 - 8:01 am
Google Ads

Google Ads In AI Overviews, AI Shopping Ads, Brand Profile Ads & More AI Ad Tools

May 22, 2024 - 7:51 am
Google Search Engine Optimization

Google & Bing Say Upgrade To WordPress 6.5 To Gain Lastmod Sitemap Support

May 22, 2024 - 7:41 am
Google Search Engine Optimization

Unusual Surge In Google Search Console Product Snippets Search Performance

May 22, 2024 - 7:31 am
Bing Search

Bing Search Testing Tag Filters

May 22, 2024 - 7:21 am
Previous Story: Google Search Console Structured Data Report Gets More Details