SEO Considerations For SOPA Website Blackouts

Jan 17, 2012 • 8:40 am | comments (13) by twitter Google+ | Filed Under Google Search Engine Optimization
 

SOPA Blackout WikipediaMany sites are taking a stand with the SOPA movement by blacking out their web site for a 24 hour period on Wednesday, tomorrow.

Wikipedia is probably the most vocal one doing this, where they wrote:

The blackout is a protest against proposed legislation in the United States - the Stop Online Piracy Act (SOPA) in the U.S. House of Representatives, and the PROTECT IP Act (PIPA) in the U.S. Senate - that, if passed, would seriously damage the free and open Internet, including Wikipedia.

This will be the first time the English Wikipedia has ever staged a public protest of this nature, and it’s a decision that wasn’t lightly made.

So you want to make a stand but don't want to lose your shirt by doing so? Google offers advice on how to blackout (turn off) your web site for a day without hurting your short-term and long-term rankings in the Google search results.

Pierre Far from Google posted the advice on his Google+ and John Mueller from Google posted them at Google Webmaster Help. Here it is and I suggest you follow them carefully if you are blacking out your web site for SOPA.

tl;dr: Use a 503 HTTP status code but read on for important details.

Sometimes webmasters want to take their site offline for a day or so, perhaps for server maintenance or as political protest. We're currently seeing some recommendations being made about how to do this that have a high chance of hurting how Google sees these websites and so we wanted to give you a quick how-to guide based on our current recommendations.

The most common scenario we're seeing webmasters talk about implementing is to replace the contents on all or some of their pages with an error message ("site offline") or a protest message. The following applies to this scenario (replacing the contents of your pages) and so please ask (details below) if you're thinking of doing something else.

1. The most important point: Webmasters should return a 503 HTTP header for all the URLs participating in the blackout(parts of a site or the whole site). This helps in two ways:

a. It tells us it's not the "real" content on the site and won't be indexed.

b. Because of (a), even if we see the same content (e.g. the "site offline" message) on all the URLs, it won't cause duplicate content issues.

2. Googlebot's crawling rate will drop when it sees a spike in 503 headers. This is unavoidable but as long as the blackout is only a transient event, it shouldn't cause any long-term problems and the crawl rate will recover fairly quickly to the pre-blackout rate. How fast depends on the site and it should be on the order of a few days.

3. Two important notes about robots.txt:

a. As Googlebot is currently configured, it will halt all crawling of the site if the site's robots.txt file returns a 503 status code for robots.txt. This crawling block will continue until Googlebot sees an acceptable status code for robots.txt fetches (currently 200 or 404). This is a built-in safety mechanism so that Googlebot doesn't end up crawling content it's usually blocked from reaching. So if you're blacking out only a portion of the site, be sure the robots.txt file's status code is not changed to a 503.

b. Some webmasters may be tempted to change the robots.txt file to have a "Disallow: /" in an attempt to block crawling during the blackout. Don't block Googlebot's crawling like this as this has a high chance of causing crawling issues for much longer than the few days expected for the crawl rate recovery.

4. Webmasters will see these errors in Webmaster Tools: it will report that we saw the blackout. Be sure to monitor the Crawl Errors section particularly closely for a couple of weeks after the blackout to ensure there aren't any unexpected lingering issues.

5. General advice: Keep it simple and don't change too many things, especially changes that take different times to take effect. Don't change the DNS settings. As mentioned above, don't change the robots.txt file contents. Also, don't alter the crawl rate setting in WMT. Keeping as many settings constant as possible before, during, and after the blackout will minimize the chances of something odd happening.

Forum discussion at Google+ and Google Webmaster Help.

Previous story: comScore Search Share December 2011: Google Up & Yahoo Down
 

Comments:

Dirty Seagull

01/17/2012 02:23 pm

I understand the protest, but authorities need to act not wikipedia

Michael Martinez

01/17/2012 06:19 pm

People need to stop participating in this nonsense.  With one exception, virtually everything that people are afraid SOPA would introduce is already happening and will continue to happen even if SOPA is defeated.  The one exception that SOPA would introduce to the mix?  Making Google and Bing accountable for the advertising they accept from companies that steal intellectual property rights and use that stolen content to make money. This goes far beyond stolen books and movies.  It reaches into the heart of every Web marketing company whose content has been scraped by sites that run  AdSense on their stolen content and who advertise in paid search for more traffic. So go ahead and support Google in dodging another $500 million fine for profiting from illegal activites, but the next time you see someone else profiting from all your hard work in the search results, DO NOT WHINE because you forfeited that privilege by supporting Google.

Travis Burke

01/17/2012 07:00 pm

Obviously there is a problem with counterfeit merchandise and stolen intellectual property on the web, and it needs to be addressed.  No one is debating that. But if you believe that the solution would be to turn control of so much of the web to the US Attorney General then your concept of freedom is much different than mine.  There is a problem that needs to be addressed, and the Congress and Senate are contemplating passing bills that will jeopardize the whole concept of a free and open internet as we know it in order to address it.  This bill must not pass!! In order to protect the property rights of a few you must believe that it is necessary to strip freedom from us all.  I can assure you that it is not. 

Afrenett

01/17/2012 09:59 pm

This bill is absolute, for lack of a better word, BS. The US government cannot even hope to achieve security levels on which piracy would be non-existent without infringing on our rights, as people of the United States. So why not help these websites in petitioning against this violation, why not put your own time and effort towards something you wish to keep? Why not advocate these blackouts, or even black out your own site(If Applicable)? I do not know a reasonable answer to these questions other than the obvious: DO all of these things; help the people of America. I hope any reasonable readers will join me in this cry for rationalism and the preservation of the rights of the persons living under the Star-spangled banner.  

Michael Martinez

01/17/2012 10:43 pm

duplicate post.

Michael Martinez

01/17/2012 10:44 pm

..."if you believe that the solution would be to turn control of so much of the web to the US Attorney General" Travis, you like so many other people who have been bamboozled by the "Stop SOPA" lies and propaganda don't seem to understand that SOPA isn't creating any new powers for the government.  They have already shut down thousands of Websites for illegal activities in the same manner that SOPA would codify.  You've already lost the war -- you're just not paying attention to the facts. There never has been a truly "free and open Internet" but there is absolutely no basis for the implication that people in the U.S. Department of Justice will immediately begin plotting the shutdown of innocent Websites if SOPA is passed.  They are only interested in sites that break the law. Innovation has never been halted or slowed by laws -- only driven by them.  That is clearly another lesson that the propagandists don't want you to learn from history. In fact, the only new aspect to SOPA is the provision for requiring advertising networks (such as Google) to sever all ties with sites engaged in illegal activity.  That means Google would not be free to continue depriving you of income by paying other people who scrape your content for clicks on Google AdSense listings. That's a good thing.

Ryan Pallas

01/18/2012 01:04 am

"They have already shut down thousands of Websites for illegal activities in the same manner that SOPA would codify." SOPA would allow the US government to shut down your ISP because someone accessed a site ACCUSED (with no fair trial or hearing on the matter) of having any duplicitous material. Your CC company could be shut down because they accepted a payment from you to a company accused of having pirated material. This means if someone accuses amazon.com of having pirated material your ISP, bank, and google ETC ALL have to refuse to let you get to amazon.com without any PROOF or any hearing. That's what the issue is.

John

01/18/2012 04:14 am

What a load of BS!  We really need a new agency deciding what we will be able to see on the internet?  SOPA, if enacted, would be disastrous.  We would then be at the mercy of the sleazy mainstream media which (if you have been paying any attention whatsoever) is total propaganda.  We are moving closer and closer to a totalitarian society.  We used to read about the Soviet block back in the 1960's.  I never could have imagined that the USA would move so far in that direction back then.  But here we are today with NDAA and now SOPA.  Don't be stupid people.  It is time to wake up and see what the heck is going on.  It's not good.

Christy

01/18/2012 05:13 am

I suggest webmasters  to use a 503 HTTP status code to tell spiders that the website is temporary unavilable.

Nunofyourbiz

01/18/2012 02:33 pm

Once again Mr Martinez cuts through the usual crap and speaketh the truth against power. Bless ya Mike! 

Michael Martinez

01/19/2012 08:11 pm

"SOPA would allow the US government to shut down your ISP because someone accessed a site ACCUSED " That is simply not true.  The bill as written specifically provides safe harbor against that kind of abuse.  You should read the bill because the propaganda has led you down the garden path to ignorance.

harsa saha

04/29/2012 02:56 am

to add : More than 162 million people saw Wikipedia's protest page

Alex

06/18/2013 10:14 am

We are taking our site down for 3 months! Is that too long a period for 503 with additional retry date set? And how should having your site closed for a month (with returning with the same content) to be handled, for not losing index rankings? Many thanks

blog comments powered by Disqus