Disqus WordPress Plugin Confusing GoogleBot

Sep 6, 2012 • 8:22 am | comments (41) by twitter Google+ | Filed Under Google Search Engine Optimization
 

DisqusDisqus has always been a challenged for Google but several months ago, they made their commenting rendering engine more search engine friendly - but many were not convinced.

That being said, recently, several bloggers noticed an unusual spike in 404 errors reported by Google Webmaster Tools. They dug into the issue and it seemed like it was coming from the Disqus commenting engine.

Here is a picture of the weird 404 errors that make no real logical sense (it should show URLs, not numbers):

Disqus Google 404 Errors

Google's John Mueller said he'd look into it but in the meantime, the webmaster emailed Disqus support who blamed Google. They wrote:

We believe these 404 crawl errors are due to search engines like Google trying to crawl dynamic JavaScript-assembled links coded in the Disqus WordPress plugin. We’ve recommended reaching out to Google directly about this, as enough feedback may help them resolve this. In the past someone from Google has had to tweak it to get them to go away, and this looks similar to the previous case.

So hopefully, the two of them will be able to fix the issue.

I have seen an unusual spike in complaints about these errors in the past week or so.

Forum discussion at Google Webmaster Help.

Previous story: Google SEO: Canonical Country Pages To Primary Page?
 

Comments:

dnyaneshwar ware

09/06/2012 12:51 pm

Hello Barry, I think Disqus support team is completely right in their explanation. Even I received such kind of 404 errors in Google webmaster tools where the broken link was coming from JavaScript assembled code. There was no anchor tag before the link, and even then Google was crawling these links. That made me think for a while if Google does crawl links without an anchor tag. That was really an insightful post.

Winson Yeung

09/06/2012 02:59 pm

I hope the disqus won't post a problem for the google search engine because I simply love this disqus commenting system. Its very neat and have good user interface and design

DaveKeys

09/06/2012 04:10 pm

Disqus combats spam big-time and that's why so many of us love it so Google is going to have to get jiggy with it.

Boost Inspiration

09/06/2012 08:01 pm

This is really happening. Showing hundreds of 404 errors in Google webmasters and increasing every day. Here's what it look like .. https://twitter.com/bstinspiration/status/243747591471390721/photo/1/large

Michael Hazell

09/06/2012 09:02 pm

Google could use the Disqus API if they really wanted to index comments from Disqus. But if you are looking for a social search engine, I would recommend Gawk.it. It came out about 3 months ago, and I have it on my site. It indexes comments and posts, which in search engine terms, is a good setup.

Michael Hazell

09/06/2012 09:05 pm

I wonder if Google uses rel="nofollow" on those links, because that might solve most of the issue. If Google still crawls those links, then Google would be ignoring that standard.

sestuff

09/06/2012 10:29 pm

I have seen Google follow nofollow links on my site. I had to block them with robots.txt to stop them from crawling certain links.

Michael Hazell

09/06/2012 10:35 pm

Were exactly is the JavaScript crafted links located at? I want to inspect and look into this further myself out of curiosity.

Nick Ker

09/06/2012 11:17 pm

Yup - I had that. Big problem if you aren't using rel="canonical" to advise the bots on what's original. I thought it was from Disqus, since I just added it recently, but couldn't be sure that the infinite duplicate content problem was not there before. A couple weeks ago I started seeing the errors. Yoast's SEO plugin wasn't marking the original post canonical so I was getting all sorts of 404s. I used plugin's redirect ugly permalinks as a temporary fix, but it seemed to mess with some legitimate query strings. A later update to WP SEO fixed the canonical problem so I think it is sort of fixed now. No new 404s in WMT anyway.

Ghassan Younis

09/07/2012 03:49 am

Is LiveFyre better, SEO speaking?

seoacer

09/07/2012 07:43 am

I can't understand Google's PR method! It never adapt my website!

Webstats Art

09/08/2012 06:50 pm

yeh blocking google is difficult

Ross McKillop

09/09/2012 02:19 pm

Same problem here, thousands (!) of 404's showing up, nearly identical to the screenshot above. Glad it's not only me, 1000x as glad that I don't have to go re-direct crazy. I'm sure (well I sure hope) Google is working on it now. Even they must have noticed the trend.

Tiggerito

09/10/2012 02:48 pm

I'm surprised Disqus aren't offering to help Google resolve this problem. Don't they want Google to promote websites using their product?

Matt Webb

09/10/2012 06:01 pm

Yup, our issues started on August 22nd and have steadily increased everyday. Commented out our disqus snippet and waiting to see what happens next. Lame...

Ryan

09/11/2012 12:10 am

We're looking into it

Ryan

09/11/2012 12:11 am

They render exactly the same way, so no. We worked with Google to follow their new standard for crawling javascript comments, but they haven't turned it on yet. We can assume Livefyre is in the same situation.

darkpollo

09/11/2012 05:39 pm

Another one with a lot of 404 erros received from Disqus since 29/08/2012. I am thinking to disconnect disqus until this is fixed...

shuron

09/11/2012 07:52 pm

Hmm. Livefyre on my site rendrs comments to plain HTML, just testes by deactivating JavaScript. A saw all the comments. So google do not need to crawl javascript.

Tiggerito

09/13/2012 09:25 am

This problem is still going on. I think I should point out that it is not the WordPress plugin at fault. I have an affected website this is completely custom built using the generic Disqus code.

ThomasDM

09/14/2012 08:56 pm

Hope they fix this soon, I'm still seeing large numbers of 404s because of this.

Nick Ker

09/17/2012 12:39 pm

Spoke too soon. Just found another wave of 404s. Now some of them are 404s from the phantom pages. Like: sitename/post/13450293848/134576876876

Nick Ker

09/21/2012 02:29 pm

Any update on this?

Ryan

09/21/2012 03:05 pm

not yet - to be clear it's not something we can "fix", it's rather something we have to work around

Tiggerito

09/21/2012 04:09 pm

Does that imply Google have changed something and you have to deal with it?

Ryan

09/21/2012 04:47 pm

That's what it's looking like, we didn't change anything on our end when it started

Michael Hazell

09/21/2012 11:56 pm

Why does Disqus even need JS crafted links?

Sebastian Norling

09/23/2012 08:05 am

I'm using the generic Disqus code on my website and I've got this exact same issue!

Augis Barkov

09/23/2012 11:59 am

I am having absolutely catastrophic decline in Google traffic. Could it be related to all those 404 Disqus errors? How are you doing, guys?

Augis Barkov

09/23/2012 12:47 pm

Well... seems that I will have to live without Disqus...

Tiggerito

09/23/2012 01:27 pm

Google said it should not cause any problems. I've not noticed any traffic changes.

Philippos

10/01/2012 08:18 pm

Here's a fix for Wordpress running on Apache. It will automatically create a 301 redirect for each of the urls containing those numbers. After you install this you can mark crawl errors as fixed in Webmaster Tools. Link to Github: http://goo.gl/HM4Zg. The plugin is submitted to the Wordpress plugin directory and currently waiting approval. Let me know if you have an issues or suggestions.

Philippos

10/01/2012 08:28 pm

Please see my comment above for a solution.

Tiggerito

10/02/2012 01:15 pm

Just thought it was worth passing on a quote from Googler John Mueller regarding this issue... "I wouldn't worry about these 404 errors -- when we discover URLs in various places, we just want to double-check them to make sure that we're not missing anything. These crawl errors won't affect the rest of your site's crawling, indexing, or ranking. If you are seeing changes in your site's performance in search, they would not be due to these crawl errors & I'd recommend starting a separate thread for those issues."

MusicFestivalJunkies

10/16/2012 05:39 am

Yikes. 45,000 errors over the last few months. That just can't be good. Considering removing Disqus...

The Big K

10/18/2012 05:01 am

Yes, It's affected our traffic too. The traffic decline started around September 4-5; around the same time Google Webmaster Team reported that the site has large number of 404 errors. It began with about 17k of them and reached to 99k.

AliBarker

12/27/2012 10:16 am

I also wish the disqus won't post a problem for the search engine optimization because I simply love this disqus leaving comments system. Its very nice and have good user interface and design.

Tiggerito

12/31/2012 01:29 pm

FYI it seems this issue is resolved. I 410ed all the disqus 404 pages and flagged them as resolved in GWMT. They have all gone.

Spencer Padway

01/10/2013 09:41 am

It is fixed: read disqus changelog 2.74 http://wordpress.org/extend/plugins/disqus-comment-system/other_notes/

Zen

10/14/2013 10:55 am

I have WP plugin 2.74 and script is 404: http://disqus.com/embed/comments/build/next/common.js This seems to be the correct one, in pattern of other disqus related script. http://a.disquscdn.com/1381441791/build/next/common.js

Mathew Porter

03/06/2014 06:05 pm

I am still getting 404's in WMT as mentioned above from the Disqus WP plugin and have only just installed the plugin on the latest build of WP, so has it just never been fixed?

blog comments powered by Disqus