Site Down Time Can Remove You From Google Index

Feb 19, 2007 • 8:14 am | comments (18) by twitter Google+ | Filed Under Google Search Engine Optimization
 

A Google Groups thread has some details that I have not seen before. If a site is temporarily down, I thought Google would not deindex the site, but give it some more time before they removed the page from the index.

According to this thread, that may not be the case.

Problem:

I've been having problems with downtime from my hosting company (am moving tomorrow as this is the final straw) and I've now practically disappeated from Google. I was in 3rd position for 5 of the more popular searches for my business and now - nothing.

Vanessa Fox of Google replies:

If the host is down when Googlebot tries to access your pages, then those pages may disappear from the index until Googlebot can crawl them again. In webmaster tools, do the pages you want indexed appear in the crawl errors section? If so, then Googlebot was unable to access them.

If you are moving the site to a new host and the pages are available the next time Googlebot tries to access them, then you should see them in the index again soon after that.

So Google may remove the pages from the index if your pages are not accessible to Google due to server issues.

But she explains a reinclusion request is not required in this case.

You're right that requesting reinclusion is only for sites that have violated the guidelines. This isn't the situation in your case, and there's no need to contact us to let us know that your site has moved and is available again, as Googlebot will keep rechecking for it automatically.

I have added some questions to the thread for more clarification.

Forum discussion at Google Groups.

Update: Besides for the excellent comments below, Vanessa replied to my questions saying:

Googlebot will try a few times before the pages drop from the index.
As for how long it takes for a page to get back in once the site is back up, that really depends on a number of factors, such as how often the site is crawled in general.

Previous story: Google Enables Pausing AdWords Ads, Keywords & Sites
 

Comments:

VinceVinceVince

02/19/2007 01:34 pm

Just a small comment... it does make sense for Google to remove the site, because it may never come back. How long should Google wait? Should they keep checking constantly, just in case it comes back in a day? A week? A month? A year? Google's users need to be able to click on the results and see webpages - every time they see nothing it reflects badly on Google - it makes Google's index feel old and unreliable. I've argued this point with a large ISP in the past who claimed that scheduled downtime in the middle of the night was acceptable.

No Name

02/19/2007 01:46 pm

I've definitely experienced this. I recently was hosted on a vps, and one Saturday it wasn't working well, on Google webmaster tools it showed 15 craw errors that day. I immediately purchased a dedicated server and began transferring the site. The whole next week I got about 200 less uniques a day (from approx ~2500/day). The next Saturday everything was improving again. Originally I had about 2200 pages in google's index, it dropped to about 1700, and now seems to be adding about 10 pages/day using "site:". I agree this makes sense from Google's point of view, the real issue for webmasters is how quickly they will add the penaltied site back.

Douglas Karr

02/19/2007 03:06 pm

I've disappeared off the index a few times (once I really was down). I would be interested to know what their algorithm was. My host does do maintenance and upgrades sometimes taking my site down for a few minutes on any given night. I have Google Webmaster tools, but even after verifying the site and requesting the sitemap be crawled, it still took a good week to get back on the index. That's a little scary! I work for an Email Service Provider and we'll reattempt Email Sends repeatedly to try to improved delivery... sometimes up to 10 times. Seems like Google should do the same. Try a site once every 8 hours 3 times... then if it's down, it's off. At least you have a few days to react, then! Seems like it would be in their best interest to find you when you're back up.

Gabs

02/19/2007 03:30 pm

Took me 4 days down time before being removed from the serps.. I was back online after 7 day.. (sorry again barry for all the bounce backs) I took 2 day once the DNS was sorted to be re-indexed.. The homepage was the only url to be de-indexed.. Hope that helps people..

Matt Cutts

02/19/2007 08:26 pm

VinceVinceVince makes a good point: sometimes temporarily down pages turn into truly-gone-forever pages, so we have to drop those pages at some point. But it's also true that we go back and revisit those pages pretty often and try to recrawl them in case the site comes back up.

Gabs

02/20/2007 09:15 am

matt would it be de-index and removed faster with a higher crawl rate.. higher crawl rate from a higher pr / linkage / changing content... ? And of course the other way around ?

Suhrud

02/21/2007 09:20 am

Recently, I had to face a situation where the domain got expired and the site was offline for almost more than 25-30 days.It had achieved good rankings.By the time the site got restored, it lost its rankings..not found in even top 100. To my knowledge googlebot didnt found my website in its last 2 visits and thus might have considered it as dead one..Will the bot revisit the site and get back the positions in search engines? Or else should I reinclude or start optimizing with less popular keywords than the one it was in rankings for and wait...Please Suggest

Jeff Glaze

02/21/2007 01:17 pm

I recently had to move my site to a new server with my hosting co. and was lucky enough that googlebot hit it just 24 hours before my site was offline for a full day. Believe me after being in the top five for 2 years, this was a nightmare. Once my site dissappeared for 3 months because my wife accidently changed our keywords from business keywords to shopping keywords. When you fix the problems, you CAN regain the position. We did. Try a search for "business networking atlanta" and atlantaevent.com should be in the top 3. I would suggest if the site dissappears from being down for some time, re-submit the url to engines once you get it fixed, that way the bots are sure to come back.

Myamar

02/21/2007 03:19 pm

I know a couple of sites that are down for more than a few months and are still doing fine on serp. Just search for 'monitorul de bacau' (a Romanian newspaper) and monitorulbc.ro is still there, the cache being from April 1st, 2006.

No Name

02/21/2007 05:48 pm

My question is : how to differentiate a website is dropped from index because of the server down, or because of it violated the TOS of google ? anyone can answer ?

No Name

03/03/2007 08:15 am

This is an interesting observation, but i have seen sites that were ceased to exist more than 1 year ago still on Google index. I think link popularity probably counts more than site downtime in getting a site indexed or removed.

beslin

03/03/2007 03:25 pm

my website is appearing in the first page in msn, yahoo, altavist etc.. but i was not able to find my site in google (searched around 100 pages) any faults? u can suggest ?

Patricia

03/07/2007 03:50 pm

Hi, I am hoping someone can help me here. I have a baby clothing website and had top placement on 10 of the top keywords for my business. We thought our domain was on auto-renew but it wasn't. We lost our domain for less then a day and a temporary page was put up by the domain registrar. Our luck, google crawled our site the day that temp page was up. Now, all of our keywords and our domain come up with an irrelevant description that was pulled from that temp page. Do you have any idea how we can fix this? Will we be dropped from these keywords even though our site is back up. That temp page was only up for about 10 hours. Pleasde help. I worked so hard on this site....

alok

12/06/2007 05:40 pm

i had submited my site also but google didnt catched it from 20 nov. i dont know why can anyone help me.

alok

12/06/2007 05:41 pm

i had submited my site www.gobananas.in also but google didnt catched it from 20 nov. i dont know why can anyone help me.

Haji

01/07/2008 05:01 pm

My site was down for a few days, but google never deindexed it.

Durbanite

01/09/2008 01:53 pm

Our server hosting various sites crashed and some sites were not accessible for a short period. Google, de-indexed the pages it tried to cache. Sites were doing exceptionally well. For those of you who were included again on the next crawl....did your regain your serp positions or did it take time to regain them?

Owen Prescott

04/15/2014 12:14 pm

My site is going to be down for a couple of days as I switch host, I really hope this will not have much negative impact to my future rankings.

blog comments powered by Disqus