Google Officially Requests Webmasters To Block Internal or External Search Results

Mar 12, 2007 - 10:45 am 3 by

Matt Cutts posted an interesting write up he named Search results in search results. In that post, he located a Google Groups thread started by Philipp Lenssen asking:

Is there any official Google statement regarding that search result on one's own site ought to be disallowed from indexing (e.g. via robots.txt)?

He cited an example of YouTube search results (a search page at YouTube) coming up in Google Web search results.

Over the weekend, Kevin Gibbons started a thread at Search Engine Roundtable Forums asking why do Google Maps results show up as indexed in the Google search results. His example search was bondi beach via Google UK, that shows this page from Google Maps listed in the search results. As you can see from the robots.txt file, Google disallows crawling of that file. So why is this page listed in the Google results? Well, it is not really listed. Someone must of linked to the page, so Google has the linkage data in the search results, which is normal on some searches.

Back to the YouTube case, Google's Matt Cutts said "YouTube added a “Disallow: /results” line in its robots.txt file." And Vanessa Fox has added a line to the Google Webmaster Guidelines stating:

Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines.

So there it is. Google does not want to show their own properties search results in the Google search results. Nor do they want your search results to come up, if they don't add value. So I suspect, I should disallow searchinternal.html from this site to show up, because at least one result has been indexed.

In some cases, site navigational elements are in a sense a search feature. You have ways to filter your results for your products, using links to do so. Let me give you an example. Footlocker.com has a Men's Basketball sneaker section. I can then refine my search by filtering down, via link navigation, to Men's Nike Basketball sneakers, I can then refine it a bit more by selecting the $50 - $74.99 price range. Are those the type of search results Google is talking about? I doubt it, because they add value.

But what about my example above of showing all my posts via internal search feature for reputation management is that bad? I found it useful, so I linked to it. Maybe other's looking for online reputation management rather see several examples from this site, as opposed to just one? I think in this case, Google would rather show one or two results and offer the "More results from www.seroundtable.com »" link, if they deem it necessary. So in this case, I should really disallow that page. But I have yet to add a robots.txt file, so maybe one day. :)

Danny Sullivan also has a great post at Search Engine Land on this topic from this morning. Danny, in part, looks at Shopping Search Engines and should Google index the search results page of shopping engines, when they don't index their own Froogle results.

Forum discussion at Google Groups & Search Engine Roundtable Forums.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 18, 2024

Mar 18, 2024 - 4:00 pm
Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Previous Story: Google's Chinese Directory Sports Features & AJAX