Google On Mobile SEO: GoogleBot & URL Structure

Feb 22, 2011 • 9:20 am | comments (46) by twitter Google+ | Filed Under Google Search Engine Optimization
 

Google Mobile SEOI get really upset when I see lost SEO opportunity from a web site structural standpoint - it bothers me. It doesn't mean all the sites I build, even this one, are 100% perfect, but it bothers me to see issues.

One area I believe I see issues in way too often is how sites deal with mobile URLs and content. I have been a strong believer on the one URL, no matter what, policy. I wrote about it in my SEO & print URLs article and I wrote about it in my mobile Sitemaps article.

I am upset when I see an article I read on my mobile change from www.domain.com/article.html to m.domain.com/article.html. It bothers me for one huge reason... I know I am going to end up sharing that URL with the m.domain.com and not the main www.domain.com and for that site, it means missed link control and opportunity. I sometimes do not understand why they simply couldn't of used the same technology to redirect a mobile device to simply serve up a different style sheet for that mobile device and leave the URL as is. It bothers me, a pet peeve, if you will.

Now I know sometimes you do not have a choice either through the CMS or budget reasons.

But when I saw Matt Cutts post a video that really guided people into using different URLs, I became sad. Really - I was sad.

So I posted a thread at Google Webmaster Help asking them to clarify why Matt would recommend using different URLs. There were other concerns, but my main concern was the URL thing. My answers have been given today.

Pierre Far from Google wrote a blog post on the Google blog named making websites mobile friendly and specifically discussed these topics. Here are the cliff notes:

  • Google differentiates between traditional mobile phones and smartphones.
  • Google has two bots: Googlebot and Googlebot-Mobile. Googlebot crawls desktop-browser type of webpages and content embedded in them and Googlebot-Mobile crawls mobile content.
  • Currently only traditional phones are supported with special useragent strings within Googlebot-Mobile, not smartphones (this may change)
  • Google said they "expect smartphones to handle desktop experience content so there is no real need for mobile-specific effort from webmasters.
  • It does not mean you can't serve a special style sheet to smartphones, Google said, "the decision to do so should be based on how you can best serve your users."
  • URL structure: For Googlebot and Googlebot-Mobile, it does not matter what the URL structure is as long as it returns exactly what a user sees too.
  • Using the same URL "is not considered cloaking by Google."
  • Mobile sitemaps: you should include only mobile content URLs in Mobile Sitemaps, even if these URLs also return non-mobile content when accessed by a non-mobile User-agent.

Based on this information, I really think you should be very conscious about the URL you serve to mobile users. If you can, I'd always serve the same URL a desktop version would get, assuming the page and content is the same, but the only difference is the format and user interface.

Forum discussion at Google Webmaster Help and WebmasterWorld.

Update: On June 6, 2011 we wrote a new post named Google: Use Single URL Without Redirect For Serving Smart Phone Content. Take a look.

Previous story: Google VPN Blocking For Geo-Specific Search Results
 

Comments:

Lauren Polinsky

02/22/2011 02:49 pm

As someone who is currently building a mobile friendly website for my company, I can tell you that the issue you talked about - being able to share the mobile friendly version or the desktop version - can be dealt with if marketers planned in advance. Most times the m.sitename.com is triggered when the device type is identified as a mobile or smartphone device. So, when providing shareable links with the Facebook, Twitter, et al... buttons, the site should share the desktop version every time. Problem solved!

Allan Gaunt

02/22/2011 02:49 pm

I was wondering the same thing with the mobile URL. Thanks for the clarification.

Barry Schwartz

02/22/2011 02:52 pm

Not if I bookmark the m.domain.com and link to it that way. I mean, it likely 301s but it loses something.

Michael

02/22/2011 07:13 pm

Can't you simply re direct?

Barry Schwartz

02/22/2011 07:14 pm

Redirects don't pass full PageRank. See http://www.seroundtable.com/archives/021832.html

Stever

02/22/2011 08:04 pm

When building mobile versions for local small biz clients I do use different URLs. I place the mobile pages in an /m/ folder instead of a m. subdomain. User agent detection is employed and phone users are redirected. Ipad users are not (their screen size is fine for full website). These mobile pages are purposely designed for the smaller mobile screen. The content is not 100% the same. In fact we try to par the content down by 1/2 to 1/3. Same core message, but far fewer words and images. Smartphone users have even less attention span than desktop/laptop users and average time on site is near half that of non-mobile users. "ok, found a local biz that does what i want and they have a click to call number right there, click". I place a link in footer of mobile version to go to the desktop version should someone want to. User agent redirects are disabled for internal navigation so mobile users don't get trapped in the mobile version should they wish to see the desktop version. If a mobile page happens to receive a link from some other site its basically just another deep internal link and that link juice gets shared throughout the site, since there are links back to the non-mobile version. an example http://showroomautospa.com versus http://showroomautospa.com/m/ Now for a website serving up the same content, say blog or news articles, then yes keep the url and just change the presentation with mobile style sheets or in wordpress use the theme switch function with user agent detection to display it in a mobile specific template, but on the same url.

Thiago Prado

02/22/2011 08:11 pm

The problem about one unique URL is that some phones( not smart) will break the website and the user wouldn't be able to see it properly. Even if you use a different CSS style, sometimes there are too many elements in the page that wouldn't be presentable in some phones.

Barry Schwartz

02/22/2011 08:44 pm

Yep, when not serving the same content, I'd do the same.

Barry Schwartz

02/22/2011 08:46 pm

that is a problem with the style sheet, not the URL.

Ryan

02/23/2011 12:46 am

I wonder where canonicals come into play with this?

online shopping uk

02/23/2011 07:52 am

Quite Interesting! That means you have to create two separate URLs one for your website and one for your mobile based website rather then using canonical URLs..

Gima

02/23/2011 11:05 am

Is just common sense, great article...

Joe Youngblood

02/23/2011 06:51 pm

Am I the only one that took away from this that Google could use mobile friendly as a ranking signal for mobile users? They clearly stated that they can tell if a page is good for a certain device/mobile os and at one point says that Google will take both urls and add them to "the correct index". It sounds to me that Google is trying to improve the mobile search experience by attempting to return websites that a user can view best on their device, this guide is like a small warning telling us webmasters to shape up the mobile experience or risk losing some mobile organic traffic.

Barry Schwartz

02/23/2011 06:54 pm

They have a special mobile index. That is nothing new. But it's for traditional phones. Not smart phones.

Joe Youngblood

02/23/2011 07:30 pm

Today's revelation about the iphone useragent might call that into question. I just did a series of google searches for phrases on websites that completely fail the W3C validation and found that websites using flash are ranking approximately 1.5 positions lower on android (not flash capable) than on my pc. anecdotal at best, but it makes me wonder. but my point was that 'traditional phones' could get a different set of results based on if those websites are friendly for them. it would be a very authentic signal to use and tough for webmasters to gauge. I can't do the same quick check as above since I don't have a traditional phone.

Stever

02/23/2011 07:30 pm

In the organic index, when searching via mobile, they add a little green phone icon next to urls that have a mobile version. This would likely increase click through rates for mobile users, particularly when competitors don't display the mobile icon. Rankings appear to be the same between desktop organic listings and mobile organic listings. So at this time mobile sites are not getting extra love in mobile search (smartphones hitting the organic index). In those organic results they will display the desktop url even when it's the different mobile url that mobile users get redirected to. I presume Google just displays the mobile icon because it knows there is some mobile specific content on the site and may or may not realize or care that there are redirects in place. You can view this from your desktop by editing the google search url from this http://www.google.com/search?q=xxx to this http://www.google.com/m/search?q=xxx Simply add an /m/ folder onto it. Now that you are viewing the mobile results in your desktop, some browsers (FF but not Chrome) will let you switch to the mobile only results from the drop down options at bottom of page. I assume that's the mobile index.

Barry Schwartz

02/23/2011 07:39 pm

Key point is searching via traditional mobile phone accesses the special google.com/m not searching on iphones or android phones.

Harish Mehta

02/23/2011 10:20 pm

ultimate ........Google info - harish mehta

Alistair Lattimore

02/24/2011 12:55 am

Barry, If you were worried about the separate URL from a sharing stand point, why not share the desktop URL from the mobile site and let their redirect system based on user agent handle the user who follows the shared link? That way if they come back to the site using a desktop browser, they get the right site. If they come back with a mobile device of some sort, user agent detection will redirect them back to the URL the user was actually viewing when they shared it. It'd also solve the link equity issue that you're referring to, in that the user is more likely to link to the desktop URL since that was what was shared in the first instance. Thoughts?

N.G.

02/25/2011 04:28 am

There's a simple solution to this, simply include an optional request parameter to return either version of the page, which will take precedence over the user agent if present.

Barry Schwartz

02/25/2011 05:47 pm

Well, I often bookmarklet items on my mobile and share stuff that way. It leads to me sharing a mobile URL not a desktop URL.

JohnSharp

02/26/2011 05:31 am

No smart phone support within Googlebot-Mobile? The viewpoint you quote - "[Google expects] smartphones to handle desktop experience content so there is no real need for mobile-specific effort from webmasters" is totally unrealistic. Great site by the way!

web video

03/28/2011 05:00 pm

Are the algorithms different for mobile though?

Bryson Meunier

05/28/2011 07:24 pm

Hi Barry, I've been reading Search Engine Roundtable since 2004, so I hope you will take this criticism in that context.  It's unfortunate that this article shows up above the fold in Google for the phrase "mobile SEO", as what you're suggesting is more mobile-friendliness than mobile SEO. I did a post on Search Engine Land on the same Google post, and the conclusions I reached are much different than what you write here: "Why Mobile Friendly is Not Mobile SEO" (http://searchengineland.com/why-mobile-friendly-is-not-mobile-seo-66192).  Your concern about duplicate content is also unfounded, as I've written before in Search Engine Land: "Don't Penalize Yourself: Mobile Sites are Not Duplicate Content" (http://searchengineland.com/dont-penalize-yourself-mobile-sites-are-not-duplicate-content-40380).  Smartphone results can differ from traditional results (14 differences here: http://searchengineland.com/14-differences-between-smartphone-search-desktop-search-results-74687), and if Google wants to remain competitive in mobile search they will need to cater to content mobile users respond to, which is content that is created specifically for them. Webmasters who format their desktop content for mobile users do not have the ability to alter their information architecture or optimize their mobile content for mobile keywords, and as research shows that mobile users search differently and have different information needs, this makes these webmasters less competitive than webmasters who build mobile sites that cater to mobile users. That being said, I do like the mobile-formatted version of your blog, and I actually read it on my Android phone because of the new look and feel. Appreciate you making small concessions for mobile visitors, even if you're not sold on the mobile user experience just yet. Best, Bryson

Barry Schwartz

05/29/2011 01:55 am

I am glad you added this comment because you make very valid points. I just personally find having one URL incredibly important because of the link structure. I am a blogger, I link out a ton and if I get a mobile URL when I am reading 3rd party content, I tend to link to the mobile URL when I write my story later on. SEO for the past 8 years or so has been about controlling your links (and of course content). I totally agree with you on the fact that there are many cases where the content, architecture and so on have to be designed differently for mobile users. In fact, this site and rustybrick.com have totally different designs, slightly different architecture but still keep mostly the same content and ALL the same URLs. You can do a lot within a smart custom built CMS to keep the same URLs and tailor the web site to your mobile users. Yes, this site doesn't do everything it could to do that but I know I can do it can be done and have the best of both worlds. Would you agree or disagree?

Bryson Meunier

05/29/2011 06:44 pm

Thanks for the response, Barry. I agree sites can be designed to keep the same URL structure, of course-- even when the information architecture is slightly different. I just haven't seen evidence that keeping the same URL is necessary when it comes to indexing and ranking mobile content. I have, of course, in my ten years as an SEO seen many sites with duplicate desktop URLs improve in ranking considerably when content was consolidated into one URL, so I'm not arguing that this principle can be valuable when it comes to individual sites, be they mobile or desktop. And I understand where your passion for the issue comes from, because I argue all the time with developers who don't understand the importance of consolidating URLs for the purpose of harnessing link equity and making the page competitive for key terms. I just think we're dealing with a new paradigm with mobile, and that certain principles that have applied in the past aren't going to be  relevant here.  One area where this is the case is QR codes. QR codes are only beginning to become mainstream in the US, but for SEOs they represent an opportunity to increase link equity by linking from the offline world. Mark Sprague did a nice write-up on the opportunity in Search Engine Land a while ago (http://searchengineland.com/qr-codes-are-you-ready-for-paper-based-hyperlinks-49684). In the past 8 years, search engines have never been able to tell that a user followed a link at the mall, and that the link delivered the information to them at the mall. Mobile visual search is another game changer. Today I can open Google Goggles and do a search without specifying a keyword. The search engine will then search its image archive for a similar image and match it with a keyword. Never in the past 8 years have these been issues, but as mobile search usage increases and outpaces desktop usage in the next 3 to 4 years (or in June 2011 in the case of Google Maps usage), SEO will change in ways that it hasn't before in order to get traffic from this brave new world of mobile searchers.  I'm devoting the next two of my Mobile Monday columns in SEL to how desktop search differs from mobile search, if you're interested. Last month I talked about the difference in search results; this month I'll be talking about keyword research; and next month I'll be looking at mobile links. Search engines have always evolved to provide a better user experience, from Florida to Big Daddy to Panda, and they will need to evolve to provide the best mobile user experience if they're going to be able to compete against potential rivals in the space. Part of that will be figuring out how to handle mobile-specific URLs when the best practice in mobile marketing is to host sites at m.*.com to minimize the number of keystrokes a mobile user needs to type. At any rate, I'm glad you're writing about this; even if we don't agree on this one issue. To me it's one of the more fascinating changes happening in search right now, and the more coverage it gets, the more likely all of us will be able to get it right. Let me ask, have you seen instances where a mobile-specific URL hasn't appeared for relevant keywords because of split link equity? Because I haven't seen it yet. In my experience it's more likely that mobile sites are not ranking because a site owner has removed them from consideration with robots.txt or canonical tags in order to avoid this duplicate content issue, ironically. Or that site owners don't include keywords and concepts on their pages that mobile users are looking for. If there are instances, though, I'd love to hear about them, as that could change the way I approach this issue.  Best, Bryson

Barry Schwartz

05/29/2011 06:53 pm

The bigger question is, would that content rank better if it wasn't split between two URLs? I am not sure - I haven't run any tests on that.

Barry Schwartz

06/06/2011 12:32 pm

Bryson, Google went on the record about this.  New post at http://www.seroundtable.com/single-url-mobile-seo-13521.html

tim dunn

06/07/2011 09:07 am

Thanks for this great article and thoughtful comments!

Bryson Meunier

06/25/2011 06:40 pm

Thanks, Barry. I responded to John Mueller in the thread.

Pali Madra

06/27/2011 07:02 am

@barry and @bryson it is inspires people like me to see such open discussions in the public by two SEO experts. For me the discussion is more interesting than the post (though the post is very good as well).

เกมส์

07/01/2011 11:53 am

Surely that'll factor into mobile SEO in the future. Given that Google is about delivering the best outcome for a user, ranking a website that they know will work optimally on a users chosen device (iPhone, Android, ..) over a standard website would make a lot of sense.

Matthew Berman

07/12/2011 07:24 am

This is exactly what I was looking for. Mobile is the way to go. Thanks. 

Alexandre

07/26/2011 03:06 pm

If we use the sames URLs for two sites (desktop and smartphones) how we can optimize them? title, h1 and other tags would be the same. It would creat a lot of duplicate content. Rel="canonical" could help in this case but the site for smartphone would be invisible in search engins for those who are looking for it in their smartphones. Right?

Alireza Sefati

07/29/2011 09:16 pm

Makes sense. From SEO prospective, I think it's just harder to put a mobile site on a different URL. It's like doing SEO all over again.  I would do a subdomain or parent URL.

Billig Seo

08/05/2011 09:56 am

But putting the content on the subdomain will separate the link juice from inboud links to 2 different places. The seo will suffer then also.

Bryson Meunier

12/31/2011 05:47 pm

John, this post is actually outdated, as Google has since offered a Googlebot for smartphone content:  http://www.seroundtable.com/googlebot-mobile-smartphone-14465.html I also object to the title of the post, as Google has expressed about eight different viewpoints on mobile SEO, as I've demonstrated recently on Search Engine Land: http://searchengineland.com/do-you-know-google%E2%80%99s-official-stance-on-mobile-search-seo-100350 It would be nice if Google can give better support in 2012 for webmasters when it comes to mobile SEO, as webmasters still have many questions when it comes to how to properly return content for mobile/smartphone/tablet users, given that Google provides separate user experiences for each.

Robert Lindsey

01/12/2012 05:16 pm

Just get a Seconds page at  http://getseconds.com (it's mobile optimized and captures all inquiries via text messaging for CRM purposes) and add a  mobile redirect. This is a simple problem for even the smallest of businesses to resolve.

Sonny

02/04/2012 06:43 pm

This has been a question on the tip of my tounge for months. It has even stopped moe deploying a mobile version of my site for fear of being penalised for duplicate content. Good to see Matt's team at Google have served some clarification.

John Rendek

03/08/2012 08:05 pm

 Yeah John that comment had me scratching my head as all smartphones cannot handle the desktop experience. So umm?

roulette hire

05/03/2012 04:18 am

why shouldn't a mobile site be m.xxxx or mobile.xxx  It makes it clear that it is a redirect to a mobile site, and google then see it as a mobile site and not part of the same website

Intellectsolve

07/19/2012 11:12 am

is it like google swithboard tag?

Saidul Hassan

08/08/2012 06:38 am

I just had to turn down a client for his mobile SEO. He doesn't have a real site while he built a mobile version! C'mon!

Howard Drive

03/01/2013 11:12 am

google gives unlimited benefit.

GN

10/07/2013 01:04 am

who's moe?

Bazinga! Web Design

03/19/2014 03:46 pm

Sorry for my ignorance but, will Google show mobile websites (m.example.com) in Desktop SERPs or just on Mobile SERPs?

blog comments powered by Disqus