Survey: Yahoo, MSN Gaining on Google

[Chris Sherman]

A new study says that Yahoo and MSN Search have made significant improvements in user satisfaction, narrowing the gap Google has traditionally enjoyed. The study also says that this could be troublesome for Google, as more than 50% of all searchers are more than willing to try an alternative engine if they don't get satisfactory results from their primary engine.

Today's SearchDay article, Yahoo & MSN Closing the Google Gap, describes the results of this new study, drawn from evaluation of more than 2,000 users and looking at more than 250 different metrics.

Comments welcome!

[Webvisitor]

As your article points out Chris the data indicates Google should be troubled by these numbers. Numbers lead to trends.
My blogs, through Blogger, rank better on Teoma (#1) and MSN(#1) than they do on Google (page 5) even though I have a dead-on keywords with NO competition. My only fault? Very few inbound links. How is pagerank more relevant? Google is not that special in our house. Everything is stuck in beta anyway! Also, why can't I just delete emails in GMail directly? Google News is far behind Yahoo News. For local I find Yahoo and even Local-Direct easier to use.
I see these numbers leading to trends that will stand the industry on its head. Your article cracked open the subject.

[pleeker]

We are finding that more and more of our clients (small businesses, mostly) are increasingly frustrated with Google both as searchers and as interested web site owners. I know others have been saying this for a year or more now, and I've been slow to agree, but it seems pretty evident that G's SERPs are not as good as they've been in the past, and the Yahoo SERPs are getting better (but still have plenty of room for improvement).

I've also been banging the drum in our own articles and newsletters for a while now that Y's Local Search is light years better than G's.

So yeah, "closing the gap" sounds right to me.

[PR Weaver]

Hi Chris,

Everything could change very quickly in the search industry!
Few years ago Altavista was leading, then Yahoo took the lead, and now it is Google.

Anyway, according this website, 2/3 of Microsoft employees still use Google

[Dave Hawley]

...are based on research with 2,000 consumers as they interacted with Ask Jeeves, Google, Lycos, MSN Search and Yahoo Search.

Personally I don't see that as being even close to a meaningful number to draw conclusions. Add 2 more zero's and I would be taking it seriously.

[Mikkel deMib Svendsen]

Personally I don't see that as being even close to a meaningful number to draw conclusions. Add 2 more zero's and I would be taking it seriously.

2000 people is a VERY large sample for almost any kind of study. If done well, and correct, the conclusions are VERy likely to be VERY precise.

From speaking to highly educated statistical engineers that I know, you would be surprised of how little samples often give you very precise results. As an example, my statistical friends say, that if you want to find out if the Danish population vote yes or not to a national election you only need 7 (of the correct selected) people in Denmark to get an answer that is over 95% correct! This has been confirmed by many such surverys and the results as they turned out.

In statistical keyword studies based on groups of keywords I found that anywhere from 0.1% to 0.5% is sufficient to get very precise results. I use it every day to estimate reach and visitor volumes for clients and I do that very well ... even with the relatively small data sample

[I, Brian]

I really like Google - always have done. Never regularly used any other search engine since I first used the net - it was simply more relevant than anything else I tried.

But...since Florida November 2003, Google has seemed more and more "precious" with its search results, yet less and less friendly to the surfer using them.

It's often seemed like Google is more intent on battling "spam" than improving relevancy, with Google presuming that both go hand in hand - when they do not. A search engine with less "spam" is not equivalent to a search engine with more relevant search results.

I don't care if the little webpage from the new website isn't an "authority" - if it gives me the relevant info required, that's fine.

Don't send me to Yahoo!, DMOZ, or About - if I wanted to search those places, I would search there first.

It's the latter practice with Google that makes myself as a surfer more open to try other search engines.

Just a personal 2c.

[St0n3y]

I don't care if the little webpage from the new website isn't an "authority" - if it gives me the relevant info required, that's fine.

Don't send me to Yahoo!, DMOZ, or About - if I wanted to search those places, I would search there first.

I completely agree with thee two statements! Only problem is that the marketshare seems to be moving back into Google's favor. When Yahoo rolled out their search it was almost evenly divided, but Yahoo has lost considerable ground since then.

[Dave Hawley]

2000 people is a VERY large sample for almost any kind of study. If done well, and correct, the conclusions are VERy likely to be VERY precise.

Last I heard there were something like 600 million internet users World-wide. 2000 is a VERY small sample and can easily mislead.

From speaking to highly educated statistical engineers that I know, you would be surprised of how little samples often give you very precise results. As an example, my statistical friends say, that if you want to find out if the Danish population vote yes or not to a national election you only need 7 (of the correct selected) people in Denmark to get an answer that is over 95% correct! This has been confirmed by many such surverys and the results as they turned out.

You mean there is 50% chance. I alone can get a 95% success of who is to win our elections here is Oz...in fact I did pick it and also picked the US elections. It means nothing.

Sorry, no way is 2000 out 600000000 isn't even close to enough to be taken serious.....not for me at least. I also bet Google don't call a global meeting over it either.

[Mikkel deMib Svendsen]

I chose to believe people with long educations in the field of statistics and surveys. I honestly don't find any valid arguments in this discussion not to do so.

[Dave Hawley]

Hey, don't take my word. What you are refering to is often called Inferential statistics

Not siding here one way or another. I remember very recently the TV net works in the US were embarrassed by early projections on how certain states would vote for the next President based on their usually accurate poles of a few voters after they had voted.

[Webvisitor]

Originally Posted by Connie

Not siding here one way or another. I remember very recently the TV net works in the US were embarrassed by early projections on how certain states would vote for the next President based on their usually accurate poles of a few voters after they had voted.

Those were proven to be biased exit polls. The pre-election polls were very accurate. Zogby, Harris, Fox all used smaller samplings, 200 to 2000 random registered and likely voters in their polling. In the final analysis these turned out to be very accurate.

[andrewgoodman]

There are no doubt some potential biases in terms of how they gather the participants for the sample, but I agree with Mikkel that 2,000 is standard and will create statistically significant results for a survey intended to represent several million people.

I do however recall from a social science stats class quite an interesting couple of weeks of argument from the professor who tried to show that commercial polling outfits try to lull us into believing small surveys like this are very accurate. It just so happens that polling organizations using weak, not-very-random sampling methods with barely enough people to be credible statistically make more profit than serious studies which really try to be random and who go after a meatier sample of 5,000 or 10,000 people.

Probably 10,000 would be a lot safer with such a large population. And I have trouble believing the sample is random.

I also have trouble believing the survey itself is a particularly good representation of real intentions or real behavior. Seems rather like some loose generalizations, like a focus group.

And did they get steak knives for showing up? Did they serve beer to the people who seemed most positive?

[mcanerin]

Garbage in, Garbage out.

Bruce Huff wrote a wonderful book I still have called "How to lie with statisics" Recommended reading, and very easy to understand.

A couple of observations:

A relatively small random sample *can* accurately predict a large random population

It's very difficult to get a truly random sample - and if you do, you may get a random sample of the wrong group.

Most "populations" that are being measured are not random - for example, internet users may have a different average education, age or income level compared to the population as a whole. So your sample must come from that group or it's meaningless.

The way a question is asked can change the way it's answered

What someone *says* they would do can be very different from what they actually do. I remember a survey back in the 1980's that followed every correct proceedure to determine what magazines people read. One key finding was that almost everyone read National Geographic and almost no one read Playb0y. Guess what the actual subscription numbers showed?

Just because there is no such thing as a perfect survey doesn't mean it's not useful or representative - you just have to understand the limitations and do your best to deal with it. And the methodology *must* be disclosed or it is useless.

Ian

[Dave Hawley]

The smaller the % sampled, the greater the chance and false/misleading results. The larger the % sampled, the less chance of false/misleading results. Statistics 101.

A relatively small random sample *can* accurately predict a large random population

It very much depends on the number of possiblities and what it is that is trying to be discovered. A small sample of people to pick a 2 horse race can often be correct. Even without other factors there is a 50% chance. Even if it's 1 person from an entire Countries population, the odds are still 50%! It has been proven many times that such "polls" have about the same succes rate as tossing a coin.

In this case we have 2000 people from a possible 600000000 people with Internet access. That is 0.000333333333333333% sample of the total with no facts on who, how, what, where & when these people were chosen. This total is spread out globaly (Sorry America) and with x number of different languages, cultures, age groups, social groups etc etc.... The potential for false/misleading results are huge and any such results should never be taken seriously.

I think Ian's point of "Garbage in, Garbage out" is very apt in this case.

[I, Brian]

This is particularly illustrative of my gripe against Google and its loss of relevancy:

http://forums.searchenginewatch.com/...1026#post31026

[I, Brian]

Here's a more illustrative example:

My memory suggests that before Florida (Nov 2003), a search for "search engine" would rank "Google.com" pretty much at the top.

Since Florida, you get sites such as Lycos and Alta Vista continually ranking higher that any Google site for that search term.

As Google has been the clear dominant search engine since 2001, the fact that Google will return an effectively dead SE like AV as more relevant for the search term "search engine", suggests what a step backwards Google has taken.

Perhaps I misunderstand something fundamental here - perhaps Google would find some pleasant spin on why they consider Dogpile as a more relevant return than Google.

However, the perception is definitely there that Google has forsaken relevancy for trying to fight "spam" - and left Google weaker on the front that made its name in search.

[Dave Hawley]

Perhaps I misunderstand something fundamental here

SE will never please all the people all the time. Ones man's trash is another man's treasure......

[pleeker]

Originally Posted by Dave Hawley

SE will never please all the people all the time.

True, but about 18 months ago, Google was coming pretty damn close. And as this survey suggests, and what my personal experience with clients and friends confirms, is that they're pleasing fewer people now than they were then.

That's the bigger issue here -- not the validity of the poll itself, or how polls work, etc. 18-24 months ago there was a pretty substantial belief around SEO/webmaster forums that Google was unbeatable. Not everyone felt that way, but many did. Do they still? Do you still? Do you agree or disagree that opportunity is staring MSN (and Yahoo) squarely in the face right now?

I'd say so, and more squarely than at any time in the past few years.