Google Deduplication Of Top Stories From Web Search Results

Jan 19, 2022 - 7:21 am 0 by

Google Vacuum

We know Google has deduplication efforts for when a featured snippet is displayed so that it might remove the snippet from the main web results. Google may do the same, in some cases, for when a URL is listed in the top stories section and then not show it in the main web results.

Google in some cases may be removing the web search result snippet when it shows that same result in the top stories section of the Google search results.

Update: Here is how deduping works for top stories:

Rest of the story continued but the update is in the tweet above.

Danny Sullivan of Google said this on Twitter after a complaint from The Verge. Danny said "it's showing in Top Stories, it is getting deduplicated from the rest of the page. Deduplication can often be useful. Doing this search in the way that user might by using solution-seeking terms rather than unusual terms in the headline, there you are at the top in Top Stories plus deduplicating means there's more variety from other publications. In searches like that, our systems also are going to generally seek to show the most helpful, reliable info they can. That's why you don't see a lot of duplicates of your article showing. Duplicates certainly exist, but it isn't that helpful to show them. That leads to headline-oriented searches. As I said before, that's super common among authors. I used to do it all the time, myself. But headline searches contain typically contain a lot of terms, so our systems shift to return pages that have those terms. his means authors are more likely to find duplicates, even though for typical searches that readers would do, these are unlikely to appear. But our deduplication feature may still kick in even for these, as was happening in this case. As I said, deduplication can be helpful. But we also understand the concern this might be raising. We've been doing this with Top Stories since last May, but we're going to revisit this to see if we should continue or perhaps make other changes. Also, I'm still checking, but I believe this deduplication is especially unique in that it only happens with Top Stories if there's a single story shown or perhaps only for the very first story shown."

So you can see, Danny is giving this as an excuse for why other publications are in the web results and not The Verge for that query. But as you can also see, he seems to explain that sometimes it does not work this way.

The interesting part is it does seem that Google is indeed deduping here when it comes to top stories.

The funny thing, hours before this, News SEO Barry Adams did a whole Twitter thread on this:

So I pointed out to Barry Adams the tweet from Danny and his response:

So I did some sample searches and Google does seem to remove a recent story from this site from the top stories and show it in web results, and at the same time, remove those stories in top stories from the web results. Notice SER is not in top stories here but is in the web results and at the same time SEL and SEJ is in the top stories but not in the web results (click to enlarge):

click for full size

But yes, the story does show in the news tab:

click for full size

But sometimes Google is not deduplicating these top stories, a query for [the hidden resignation] shows the Business Insider story in top stories in some browsers twice and some once (thanks Glenn Gabe for the query and discussing it with me):


click for full size

Not Deduped:

click for full size

Here is another example from Glenn:

So it is not clear when and why Google might deduplicate a URL from showing in web search when it also shows in top stories. It might be a timing issue or something else.

Danny Sullivan did say "Also, I'm still checking, but I believe this deduplication is especially unique in that it only happens with Top Stories if there's a single story shown or perhaps only for the very first story shown."

We hope to get some clarity soon.

Forum discussion at Twitter.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: February 26, 2024

Feb 26, 2024 - 4:00 pm

Google To Make A Content Deal With Quora Next?

Feb 26, 2024 - 7:51 am
Google Maps

Google Reviews Disclaimer For Beta Questions

Feb 26, 2024 - 7:41 am

Google Most-Read Articles Search Carousel

Feb 26, 2024 - 7:31 am
Google Ads

Google Ads Tests New Change History Page

Feb 26, 2024 - 7:21 am
Web Analytics

Google Analytics Referral Spam From Poland - Complaints

Feb 26, 2024 - 7:11 am
Previous Story: Recipe Schema Markup No Longer Supports Ranges For Cook, Prep & Total Time