Google's Matt Cutts: Bing It On Challenge Was Obviously Flawed

Oct 2, 2013 • 8:53 am | comments (59) by twitter Google+ | Filed Under Bing Search

Bing It On ChallengeA little over a year ago, Microsoft launched this campaign named the Bing It On Challenge, which was basically a blind-search taste test pinning Bing against Google, while removing the branding.

Ian from Freakonomics reports that the challenge was not just a "little fishy" but he feels Google may have an "deceptive advertising claim against Microsoft." He details why in his post at Freakonomics.

Matt Cutts, Google's head of search spam, shared this on Google+ and added some more color to his own thoughts on this challenge.

I have to admit that I never bothered to debunk the Bing It On challenge, because the flaws (small sample size; bias in query selection; stripping out features of Google like geolocation, personalization, and Knowledge Graph; wording of the site; selective rematches) were pretty obvious.

It is funny, because when I posted a poll on Who Won In Bing It On Challenge? 60%+ said Google won.

Anyway, well worth a read when you have time.

Forum discussion at Google+.

Update: Here is a statement from Microsoft PR, and also make sure to check out this comment from Microsoft's Matt Wallaert.

The professor’s analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100% accurate and we’re glad to see we’ve nudged Google into improving their results. Bing it On is intended to be a lightweight way to challenge peoples’ assumptions about which search engine actually provides the best results. Given our share gains, it’s clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service. - Attributable to Matt Wallaert, Behavioral Scientist, Bing

Previous story: Here Come The White Google Navigation Bar Complaints...


Brett Dixon

10/02/2013 12:58 pm

I hated the whole "trash talk" a competitor thing from the start. Shouldn't be done by anyone full stop. It's cheap.


10/02/2013 12:58 pm

ok so we have had 15th birthday and oh umm hummingbird and this is what Matt chooses to finally talk about. Seriously Matt.

Thom Craver

10/02/2013 01:01 pm

While the results of Microsoft's polling company may be inaccurate (I'd hesitate on actually labeling it "deceptive"), your non-scientific poll says a lot. If only 60% of the people think Google's results are better, that's still nearly half the sample that thinks Bing's results are just-as-good or better. Even if it isn't a 2-to-1 as Bing claims, it's still much higher than most of us give them credit for. For a company struggling to get to 20% market share, when 40% of a user base (sampled) thinks the product is on par with the biggest competitor, you have to wonder why everyone's always so against them.


10/02/2013 01:02 pm

I feel google will win the game. Google has given more valuable service to all people in all over the world than bing


10/02/2013 01:07 pm

You have to understand the audience who took the poll. Read the comment sections and understand why a comparison of Google vs. Bing based on the opinions of those posting here is absolutely swimming in confirmation bias. Besides, us paid-Google-shills are so few in number.

Barry Schwartz

10/02/2013 01:10 pm

my poll didn't claim it was deceptive, freakonomics did.


10/02/2013 01:17 pm

That and sabotaging competition products by making them not work with any MS product are pretty much hallmarks of Microsoft. They have been doing it since the 80's.


10/02/2013 01:26 pm

This is so predictable ) When the company/government/any structure doesn't want to acknowledge their own faults, it starts to point to others. Who is better : USA or Russia, Columbia or Venezuela, communism or capitalism, my family or neighbors ......... Common Matt, you made poor product. Comparison will not make it better. Work on it instead of pointing to others.


10/02/2013 01:41 pm

I should include the fact that my opinion, like every else, would be subject to the same confirmation bias. It's a natural tendency so understanding the, in this case, audience who voted, is important to drawing meaning from the result. Especially when the poll has to do with something everyone here obsesses over and has strong opinions on. Nothing wrong with that, but thought it worth pointing out.


10/02/2013 01:45 pm

I took the challenge and saw a lot of domain crowding on Google's results whereas Bing was more diverse. I performed the same query outside of the Bing It On site and results were similar. I personally prefer more choices and Bing continues to provide that.


10/02/2013 01:48 pm

I would like to assume that like all companies, they are both working on it and deflecting it at the same time. But Matt didn't write the article in question and was responding to something that, if in his shoes, any of us would love to share with a wider audience. When Freakonomics posts something about your company (and about a part of the company you are intimately involved in), it would be strange to not want to share it (Matt was posting on G+ after all). If they featured something positive about my company, I would share it all around...because Freakonomics is a gem of a book/site/radio show.

Stephane Brault

10/02/2013 01:57 pm

It seems to me that Bing is investing more into marketing its search engine rather than improve it.

Yo Mamma

10/02/2013 01:58 pm

Matt Cutts is a sore loser and looks good on a losing side. Cutts must stay with Google all the way until its on the bottom of the ocean. I personally haven't bothered to debunk Google's corrupt search results. I just know it won't last.

Yo Mamma

10/02/2013 02:00 pm

I am not Freakonomics nor do I give them credibility for anything they may say about me, but if they did, I wouldn't bother responding. Matt Cutts has nothing better to do than watch his ship sink. And that I enjoy with him very, very, very, very much

Yo Mamma

10/02/2013 02:01 pm

Everyone has an opinion. Just like a rear end. You may like yours. I may not

Yo Mamma

10/02/2013 02:03 pm

No, they're just looking for a good reason to sue Google. And I give them my blessing

Stephane Brault

10/02/2013 02:04 pm

Why would they sue Google?

Yo Mamma

10/02/2013 02:04 pm

Because it will be good for world karma


10/02/2013 02:07 pm

I am also curious as to how you read this as Bing looking for a reason to sue Google. What?


10/02/2013 02:08 pm

The "Bing It On" isn't about what company is better, is about what search engine gives better and more accurate search results to its users, of course, Bing is the winner, Google SERPs are boring and monopolized at this moment.


10/02/2013 02:16 pm

Manufactures, who are confident in the product they made, who are proud of the product they made don't bother to share. Why share when it's obvious? Did you hear from Apple : "Hey, watch, Washington Post published an article about the iPhone .... hurrah!! " The perfect product would talk for itself.


10/02/2013 02:23 pm

An individual within the company vs. the company itself (or at least through the official channel of communication that counts as 'the company') responding to something are different. I have shared positive articles about the company I work for with people in my social network. I have seen many of my friends do the same thing. Matt shared it on his G+ page which is basically him sharing the article and his opinion to his social network. Maybe I am a corner case, but I think the distinction between an individual sharing something positive from an external source and a company (as a whole) doing the same begs the distinction.


10/02/2013 02:46 pm

it beats Google in one thing - less scrapers and pirate sites on page 1 for most queries.


10/02/2013 02:52 pm

When Washington Post mentioned my website I also called everyone I knew. But neither my website nor your company is one of the biggest on this Planet. Different levels of responsibility, quality, attention, investments involved ... You can't compare companies with 50 and 44,777 employees. I, being the loyal consumer of Google search, would like to see the head of spam team working on spam problems, not sharing articles. Or Google doesn't have problems with spam ?? Or articles about Google are published so rare, that any new should be featured no matter what? P.S. Just tried to find the number of Google's employees. Typed into the search : how many employees google has Knowledge Graph ??? Hummingbird ??? Smart search ??? Try by yourself and see how much time will it take to get such simple piece of information about Google using Google search. And then tell me - is it time for Google employees to share articles ?


10/02/2013 03:07 pm

I see your point about the size of the company, but I am confused as to your conclusion. Wouldn't a much larger company have a higher chance of someone within that company sharing some positive news with their social network(s)? I would argue that a company with 50 employees is much less likely to have someone from within the organization post about news to their social network than a company that has 44,777 employees. There are just more people to potentially share. Again, it is a matter of, in my opinion, individual action, and not action through official company channels. Wouldn't more potential actors translate into more potential social sharing about the company? About your objection to the head of search spam working on problems vs. posting an article...I don't see how they are mutually exclusive. Matt has time after work, time during lunch breaks, time on flights, time during functions when he is not actively participating (waiting to present and using his smartphone for instance). The post this article is about was put up 4:50pm. The man basically lives online...why wouldn't he have time to make a quick post? I really would like to know why you think they care mutually exclusive (or at least seem to imply they are by your comment).


10/02/2013 03:14 pm

Sorry, forgot to address your 'PS': I entered, "How many employees does Google have" and the first result was the answer I needed. Not only did it link to the current employee headcount, but it is broken down by year and quarter (up to Q2 of current year). It also breaks the headcount down by Google stand alone and Motorola stand alone numbers. The next result was a Wikipedia article about Google, and the headcount is listed in a table right on top of the page (using the same financial tables as the first SERP result to populate the latest employee count). It took me very little time to find this it was listed in the first two ranked results.


10/02/2013 03:16 pm

People still use Google? Wow! They must have a boring life courtesy of white spaces. lol


10/02/2013 03:22 pm

I gave up with Google. Even if you make the effort to adjust or make a deeper query, the same 10 companies are being showing on the first page. Google has become too superficial. zZz zZz... DuckDuckGo is what I use now.


10/02/2013 03:42 pm

Very little time ? It should be flashed into your eyes immediately, like they do if you type-in GOOG. I picked up "how many employees google has" from the drop down menu GOOGLE SUGGESTED to me. The first result requires me to dig their financial sheets. Then bunch of the Q&A websites with the replies posted more than a year ago. Then Wikipedia with the HUGE article I have to observe to find this tiny string where the number is posted. 5-7 minutes at the best !!!! And this is simple query about the company itself. Now imagine how much time do you waste if you do tens of queries per day. Up to an hour of YOUR LIFE is stolen by Google each and every day !!! Did it bother me 3+ years ago? No. We all have been working. There were mistakes, but it's normal while people do work. Why does it bother me now? Because of the endless reports of success. Panda - success Penguin - success Knowledge Graph - success Hummingbird - success Now Matt Cutts points to another story of success. 5-7 minutes to get number of the Google employees - is this the success ?? Some of the top results provided to me in the unknown languages - is this the success ? 10-15 minutes to get information that Stephen Hawking didn't get Nobel Prize - is this the success ? Well obviously we have different understanding of success. But I'm THE CONSUMER and I believe, that I know better how does the successful product look like. There is no reason to show me articles. Just make good product and I acknowledge your success by myself.


10/02/2013 03:44 pm

Sorry, have to work. I have only 24 hours per day. Obviously Matt Cutts has more )


10/02/2013 03:45 pm

Except that surveys show a major brand bias towards Google. Not a results set bias: Also, I prefer Bing over Google because it doesn't have the knowledge graph, which is now making me search twice in same cases for an answer, and I can CONTROL the personalization, it is not forced on me.


10/02/2013 04:50 pm

Doesn't the actual Bing ad say something to the effect of "Bing won 2-1 on selected searches"? It's the qualifier "selected" that everyone ignores and is the genius of the ad. They never state that users prefer Bing to Google for ALL searches but people only hear what they want to hear. And, please, if it's search quality you want and not all the pretty bells and whistles, duckduckgo blows them all away!

matt wallaert

10/02/2013 06:58 pm

One additional consideration is actually homophily: the tendency to be friends with folks who are like us. Given that the people reading SRT are likely in the technical elite, there is a decent chance that the technical elite (which may have a strong confirmation bias) is overrepresented in our friend samples. (Note: I work at Bing.)

matt wallaert

10/02/2013 06:59 pm

(Note: I work at Bing, so take all this from that viewpoint.) A couple of notes are important before I talk about the claims made. There are two separate claims that have been used with the Bing It On challenge. The first is “People chose Bing web search results over Google nearly 2:1 in blind comparison tests”. We blogged about the method and it was used back when we kicked off the Bing It On campaign in September 2012. In 2013, we updated the claim to “People prefer Bing over Google for the web’s top searches”. Now, on to Ayers' issues and my explanations. First, Ayers is “annoyed” by the sample size, saying that 1,000 people is too few to obtain a representative sample on which to base a claim. Interestingly, he then links to a paper he put together with some students, in which he also used a sample size of 1,000 people. He then subdivided the sample into thirds with different conditions and still manage to meet conventional statistical tests using this sample. So I’m confused: if he got significance, is it hard to understand that we might? A sample of 1,000 people doing the same task has more statistical power than a sample of 300 people doing the same task. Which is why statistics are so important; they help us understand whether the data we see is an aberration or a representation. A 1,000 person, truly representative sample is actually fairly large. As a comparison, the Gallup poll on presidential approval is around 1,500 people. Next, Ayers' is bothered that we don’t release the data from the Bing It On site on how many times people choose Bing over Google. The answer here is pretty simple: we don’t release it because we don’t track it. Microsoft takes a pretty strong stance on privacy and unlike in an experiment, where people give informed consent to having their results tracked and used, people who come to are not agreeing to participate in research; they’re coming for a fun challenge. It isn’t conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it. So we simply don’t track their results, because the tracking itself would be incredibly unethical. And we aren’t basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific). And on a personal side note: I’m assuming that the people in Ayers' study were fully debriefed and their participation solicited under informed consent, as they were in our commissioned research. Ayers' final issue is the fact that the Bing It On site suggests queries you can use to take the challenge. He contends that these queries inappropriately bias visitors towards queries that are likely to result in Bing favorability. First, I think it is important to note: I have no idea if he's right. Because as noted in the previous answer, we don’t track the results from the Bing It On challenge. So I have no idea how using suggested queries versus self-generated queries affects the outcome, despite his suggestion that we knowingly manipulated the query suggestions, which seems to be pure supposition. Here is what I can tell you. We have the suggested queries because a blank search box, when you’re not actually trying to use it to find something, can be quite hard to fill. If you’ve ever watched anyone do the Bing It On challenge at a Seahawks game, there is a noted pause as people try to figure out what to search for. So we give them suggestions, which we source from topics that are trending now, on the assumption that trending topics are things that people are likely to have heard of and be able to evaluate results about. Which means that if Ayers is right and those topics are in fact biasing the results, it may be because we provide better results for current news topics than Google does. This is supported somewhat by the study done to arrive at the second claim; “the web’s top queries” were pulled from Google’s 2012 Zeitgeist report, which reflects a great deal of timely news that occurred throughout that year. To make it clear, in the actual controlled studies used to determine what claims we made, we used two different approaches to suggesting queries. For the first claim (nearly 2:1), participants self-generated their own queries with no suggestions from us. In the second claim (web’s top queries), we suggested five queries of which they could select one. These five queries were randomly drawn from a list of roughly 500 from the Google 2012 Zeitgeist, and participants could easily get additional sets of five if they didn’t want to use the queries in the first set. There is just one more clarifying point worth making: Ayers' noted that only 18% of the world’s searches go through Bing. This is actually untrue; because Bing powers search for Facebook, Siri, Yahoo, and other partners, almost 30% of the world’s searches go through Bing. And that number is higher now than it was a year ago. So despite his assertions, I’m happy to stand by Bing It On, both the site and the sentiment. (For those who have them, I’m always open to questions about the studies we conducted – feel free to shoot me a note or leave a comment. This invitation is also open to Ian and his students.)


10/02/2013 07:06 pm

I love all these posts on SEO sites that have comments stating they don't use Google anymore, yet when market shares come out nothing changes. Google owns 80+ percent every time. We need to stop living in our little SEO bubble.


10/02/2013 07:55 pm

All because bing made up a better slogan than google......... BING is a better search engine im finding...... Iv always preferred the bing image results over google's...


10/02/2013 08:21 pm

yes, bing really cheated us. They gives to gugle huge number of searches (more than in reality). 60% of googlers looks like not agreed. bing have all chances to win now, google is broken for 2 years. they rate content only by links, and mr cutts seo ideas but not by real quality!


10/02/2013 08:30 pm

yes, you need to provide good competition to google, because they now clear dictators. In many areas bing search better than in current google. Bing not rewrite my queries to provide full non-sense answers. BTW: check any deep query on google about programming and you will see it instantly what results is crap and ads above fold. In the bing i get more sites and better results. Also I love to see small sites, not only wikipedia, youtube & amazon monster sites!


10/02/2013 08:34 pm

I wish you more reports like this one. As far as I know, your campaign never pretended to be scientific. For me, it was more like a game level. But if it fired nearly scientific discussions, then, I suppose, it worth to pay more attention to Bing. Perfect promotion ) Congrats on the job done )


10/02/2013 08:35 pm

rest in peace, panda & penguine, and new google hembird something.


10/02/2013 08:38 pm

google love google & own sites, so looks like wrong selection of query. ask better "how to fix google"


10/02/2013 08:39 pm

lets him write "the best content". :)


10/02/2013 08:40 pm

you think so, mr. world?


10/02/2013 08:50 pm

most easy example of google quality. search google for p-o-r-n and you will see crap even with crappy in-depth articles (total nonsense). It what they call "everything for user".


10/02/2013 10:00 pm

Well said joey!

Yo Mamma

10/03/2013 01:11 am

Slam dunk - Chew on that one Matt Cutts. Google is a sinking ship! See ya at the bottom of the ocean

Para Friv

10/03/2013 01:30 am

I wonder about this content, I thank you for providing information and wish you happy.

Morgan Akchehirlian

10/03/2013 08:32 am

Agree that you are right to some extent but from design/User Interface to all updates that you see are for better user experience.Their is still lot of spam and manipulation in rank but Google is still trying to find out ways to bring better results. Google love text and this is the reason why I think text based sites still achieve better ranking than media players(videos) sites.


10/03/2013 12:18 pm

google have empty domain entry page (no content), need to penalize it! who is penguin or panda today? :-)

Thom Craver

10/03/2013 01:09 pm

I never claimed you said that. You quoted the article, mentioned both Ian and Freakonomics. I was making comments at large about the article that happens to be on your site. In my comment, I called it inaccurate, suggesting that I - unlike Ian - wouldn't call it deceptive. Why are you guys always so defensive?

Barry Schwartz

10/03/2013 01:11 pm

I am not defensive!!!!!! ARGH! ;-)

Charles Floate

10/03/2013 02:24 pm

Hence why I don't have an Xbox and used a crack version of Win.8 and Office


10/03/2013 02:59 pm

That might not be the wisest thing to announce publically mate :) Having said that if OpenGL 2 and Linux gaming takes off like it looks like it will, I'll be ditching windows as well :)

Charles Floate

10/03/2013 03:01 pm

YOLO I suppose xD + Me announcing isn't evidence enough for anything :P Oh I'm a big Ubuntu fan anyway ^.^


10/03/2013 04:27 pm

Matt, you are doing a terrific job at Bing, I'm falling more in love everyday with your search engine. The best you can do is to no copy any of the stupid and monarchical Google ideas and guidelines, they are getting worse and boring with every update and refresh to its dumb algo, it is a waste of time trying to find something in Google since its SERPs are being monopolized for the same 5-10 companies. Thanks. :)


10/03/2013 10:05 pm

and for small business also :)


10/03/2013 10:10 pm

yeah, my congrats, matt for your new pet. just a bird life not for long, based on information from wikipedia. xoxo


10/04/2013 04:35 am

The goal of Bing It On Challenge" was to make people aware of what Google was doing with search results. Google will not clean up its acts unless there is a challenger

Jitendra Vaswani

10/04/2013 05:35 am

nobody can sue google :)

blog comments powered by Disqus