When To Use 404, NoIndex & 301 Redirects In Farmer/Panda Fix

Mar 14, 2011 • 9:03 am | comments (10) by twitter Google+ | Filed Under Google Search Engine Optimization

Google Farmer/Panda FixThe Google Farmer/Panda update hit and 40% said they were impacted by the update.

We documented advice on how to fix the issue, if you were one of the sites impacted by this update. But specifically, what do you do with the low quality content?

Do you 404 the pages so they are gone forever? Do you 301 redirect those pages to higher quality content pages? Do you noindex the pages? The answer is it depends.

Google's JohnMu made it pretty clear in a Google Webmaster Help when you should use a 404 vs a noindex vs a 301 redirect. In short, 404 the pages that you don't plan on fixing. You can noindex the pages that are currently low quality and when you improve the quality of the content on those pages, you can remove the noindex. If you have other pages you can redirect the low quality pages, then use a 301 redirect.

Here is how John put it:

  • Completely remove all pages that you absolutely don't want anymore. Let them return 404 (and make a great 404 page so that your users can get to where they were headed, or find something related). See http://www.google.com/support/webmasters/bin/answer.py?answer=93641 Yes, those pages will show up as crawl errors in Webmaster Tools, but that's fine -- they're supposed to. They won't negatively affect the rest of your site's crawling, indexing or ranking. Having pages that return 404 is fine and to be expected. Using a 410 ("Gone") HTTP result code may be a tiny bit faster, but overall you don't have to worry about the difference, a 404 is ok.
  • If you have products that you can merge, then use a 301 redirect from the alternate versions. If you can't use a 301 redirect, use the rel=canonical link element.
  • If you have entries that you want to rewrite, then using a noindex meta tag sounds like a good solution. I'd keep the URLs in the Sitemap file (you want Google to recrawl them so that the noindex meta tag is seen) and make sure that they're not disallowed by the robots.txt file. As Cristina mentioned, this means that your site will still need resources to handle the crawling of those pages, but I assume since we've crawled them in the past, that load is not a problem, right?

Once this is done, John said it still might take some time for the changes to take effect in Google. John said:

Keep in mind that any changes which you make in this regard are going to take a while to affect the algorithms. We'll have to crawl and index your updated (or removed) content, and that can take a bit of time. The better your site's URL structure is (minimal duplicate content & easy to crawl), the faster we'll be able to update things, but I certainly wouldn't expect any overnight changes. It took a while for your site to get where it's at now, and it'll take a while for those updates to be visible. On the other hand, that also gives you time to really revamp your content & site structure too :-).

There is some really solid advice in the thread.

Our ongoing coverage and stories on the Content Farmer/Panda update:

Forum discussion at Google Webmaster Help.

Previous story: Google Says Ads Don't Play Big Role In Ranking Algorithm



03/14/2011 10:56 pm

We have noindex/nofollowed most of the content that could be considered content farm and removed all the links to it and it seems to work! After some days our ranking dramatically increased even better than before the Panda/Farm update. We decided not to remove the content completely since even if it was sort of low quality many sources were linking to them, so 404 those incoming links seemed like a waste.


03/15/2011 01:17 am

Just to avoid any misunderstandings, "404 - Page not found" can also indicate that the page is temporarily unavailable. But "410 - Gone" means that the page is permanently removed.


03/17/2011 04:57 pm

Hey Barry, Found one recovery story. http://www.labnol.org/internet/google-traffic-after-panda/18914/ It might be that March 10th blip some people reported.

Barry Schwartz

03/17/2011 04:58 pm

Yep. I tweeted it this AM. So far one.


03/17/2011 05:12 pm

Only a couple thousand to go.... One thing I wonder is if that official Panda thread on Google Forum is the largest thread ever. It's at 836 replies right now, the largest I ever remember seeing was mid-300s.


03/19/2011 06:14 am

Just to clarify please, did you tag the low quality content as noindex/follow? Did you also nofollow the internal links on other pages that point to the these pages? Did you see a traffic dip after this implementation before you saw the spike How many days did it take for the recovery? Many thanks!


03/19/2011 08:02 am

What do you mean by tag? The posts we considered low quality were blocked by the engines by the noindex/nofollow header tag and additionally blocked in the robotos.txt... This were articles we posted that aren't specifically related to our "important" content... Let's call it general stuff... All the internal links were deleted, apart from the "previous and next post" links that we couldn't block since the site is based on wordpress... The SE traffic was almost nothing before the change so can't say if there was a dip... About a week... You're welcome. There is one major factor that I didn't mention in the previous post, at the same time the "low quality" content was blocked we started a "facebook campaign" and got a few thousand likes in a short period of time, which might have helped the recovery in the SEs.


03/19/2011 11:17 am

@samuelpeterson Quick questions were FB likes pointing to a) your home page (root URL) b) FB Fan Page C) Deep Inner Links. Just curious.


04/13/2011 07:06 am

Since the first wave of Killer Pandas were let loose I started reviewing old content and removed a fair amount. Now I have been hit by the second ROW wave and left wondering what is left to remove. Everything I have written I naturally want to keep. Even those shorter articles have a point. If they were longer it would be waffle, if they were not there some small insight would be lost forever. What to do? I am reluctant to go chopping away at a site I have spent 5 years slowly building when I do not even know if or which articles could be the problem. Crazy days.

acheter carte ds one

04/25/2011 08:10 am

IPhone continues to collect new data even if canceled http://bit.ly/fXYHjW

blog comments powered by Disqus