The Effect of Brand Awareness on the Evaluation of Search Engine Results

[NewKidOnTheBlock]

How do you guys feel about this study?

http://live.psu.edu/story/24878

Can anybody with a statistics background (or well the knowledge necessary) tell me how insignificant this study would be? especially considering the 32 participiants probably only saw one of the queries on each of the "search engines" (as they probably would have caught on if they had seen the same results on each of the 4 engines four times in a row!)

What conclusions could really be drawn from that..?

[beu]

Originally Posted by NewKidOnTheBlock

How do you guys feel about this study?

http://live.psu.edu/story/24878

Can anybody with a statistics background (or well the knowledge necessary) tell me how insignificant this study would be? especially considering the 32 participiants probably only saw one of the queries on each of the "search engines" (as they probably would have caught on if they had seen the same results on each of the 4 engines four times in a row!)

What conclusions could really be drawn from that..?

When a "search engine created in-house with no brand-name recognition" earns the highest score for 25% of terms in your brand awareness study, you should know there is a problem!

Here are a few:

Who were the 32 participants and what was their initial perception of the brands in question?

Did they use the term "laser removal", "laser removal," or "laser removal."?

How can you say that a "study ties branding not just to product identification but also to product performance" using the "product" of a single brand in an environment where there is no "product performance"?

My favorite part of this "study" is "MSN Live Search" doesn't seem to be a brand!
http://www.microsoft.com/library/too...rks/en-us.mspx

"LIVESEARCH" is a trademark owned by someone Albuquerque, NM.
http://tarr.uspto.gov/servlet/tarr?r...entry=78882305

[NewKidOnTheBlock]

AI2RS, the search engine created in-house with no brand-name recognition, fared the worst. The researchers calculated its average precision rating as 10 percent below the average although AI2RS had the highest score when the query was "laser removal."

I only saw now that it fared the best for laser removal...

However..other than the fact that we dont know who the 32 participants were, etc. I find the single biggest problem with this study to be that 32 participants can by no means be a statistically significant sample size. And jumping to conclusions from a(n extremely) small sample size is hardly ever a good idea.

And making Google out to be perceived as average because they only scored "0.7%" above average and thus making them out to be average and this fact to be very surprising is silly imho, as in a representative sample size that could also mean they're way above average or way below average.

[beu]

I only saw now that it fared the best for laser removal...

Right, one of four or 25%!

[SkyLite]

It's a classic and "Perfect" example of how "Brand Heads" do testing. Put 34 people in a room and see if you get some data to support the last 3 years of college marketing classes your parents paid for at "Bland U"

Besides.....I'd vote the same way. I like Google Results. I try Yahoo now and then but no dice. So I sit in this room and they tell me these randomly selected search results......no......all GOOGLE RESULTS....are from Yahoo and from R2D2 Search. I've no background with R2D2 but if these are Yahoo results...man they are kicking butt! Compared to normal....these have caught up and maybe even beaten Google themselves! Good for them!

I wonder if we randomly looked in on the Marketing classes at this school and counted the students. Would the number be around "32"?

[cryptblade]

Interesting... probably already beaten by others, but I find it interesting that NO marketing agency, NO marketing research, or any other marketing/interactive marketing resource picked up this new press release by PSU.

A quick search for the title of the study shows some "science" sites.

Apparently - the only ones interested are academic folks because the size, scope, and conduct of this study was nothing more than academic.

What's interesting is that this research was conducted by Penn State's College of Information Sciences and Technology (IST). At the risk of offending Penn State grads here, I went to Carnegie Mellon University - well known for computer sciences and engineering (and Lycos fame). I've never heard of anything cutting edge in information sciences coming from Penn State.

But I digress. Clearly, this was an academic study for the college itself - not even meant to be a statistically relevant marketing study. Otherwise, I believe Penn State's business and math/stats majors would absolutely "E-nnihilate" (-thank you Mohammed Ali for that one) them.

Seemed more like an attempt to get attention for the school - but, in the words of Shakespeare (and Faulkner), "full of sound and fury, signifying nothing."

[beu]

Oh no, don't let "we are Penn State!" hear you say that!
http://forums.searchenginewatch.com/member.php?u=76

[cryptblade]

All the props in the world to Mr. "SEOBook". But my opinion still stands. Also did Aaron study at the IT school or something else???

I didnt study at CMU's CS or CIT or...whatever... I studied at CMU's Business School...and I studied Stats..and Marketing Research...oh yeah.

[beu]

he he he, no idea... just thought it was funny!

[cryptblade]

Even with stats and marketing research, I learned the ultimate truth about stats from Mark Twain - aka Samuel Clemens:

There are three types of lies: lies, damn lies, and statistics.

OOOOh baby.... the wisdom of the ages....

[NewKidOnTheBlock]

There are three types of lies: lies, damn lies, and statistics.

I have to say I only find this true, b/c people are susceptible to lies in the form of statistics.

Who made this statement saying that statistics would be as important for citizenship as reading and writing are these days?

Maybe the guy exaggerated a bit, but I think if you know like just the very very basics of statistics, you can tell if something is crap...and everybody should know about them. It's as useful as mental arithmetics are.

I have to say I dont like the fact that statistics gets a black eye for the fact that most people dont get the very very basics of statistics and thus can be fooled by so many silly studies...or let me put it differently: I dont think its the peoples' fault but it's the education system that fails here.

Just replace one of those theoretical maths classes that are of little use to most high school students (of which I had tons in school) with a practical statistics for beginners class. Just ONE...and most people wouldn't be fooled by biased data or tiny sample sizes anymore...

(but then again maybe in us high schools you are taught stats already? over here we're not..)

[beu]

"Lying with statistics" is quite popular!
http://www.google.com/search?q=%22ly...+statistics%22

On a side note, this topic reminds me of the questions posed in a book that Google gave all of it's employees a few years back. The book is called "Freakonomics and it answers questions like:
"Which is more dangerous, a gun or a swimming pool? What do schoolteachers and sumo wrestlers have in common? Why do drug dealers still live with their moms? How much do parents really matter? What kind of impact did Roe v. Wade have on violent crime?"

http://www.freakonomics.com/thebook.php

[cryptblade]

Ah Deutschland and your educational system. I read about it in high school - the entrance exams and stuff.

Anyway - I would agree with you with what you said. But, the problem is that the people LEARNING the stats needs to actually understand the big picture. That is, what context the stats and study was done.

Basic stats talks about how to analyze the numbers - not the context that those numbers fit.

It's not statistics that is really the fault. It's ALWAYS the interpreters. In this case, this accursed Penn State College of Whatever - and their faulty interpretation.

A better way for these guys to more accurately represent their study would be to show - hey, we at Penn State built a similar search engine but it wasnt well received. We think it's due to branding.

But instead, they made the leap and JUMPED to a conclusion for all the rest of us.

Unfortunately, the word "statistics" gets all bunched up in along with the interpreters and gets the black eye. But the black eye is due to the interpreters of stats.

Big big challenge to actually educate people to take things contextually. But then again...if people were educated to take things contextually - and to have wisdom - then democracy wouldn't succeed.

[cryptblade]

I actually heard of this - based on the questions not based on the author or the new word (what is that called - neologism?).

Anyway - I bet it would be interesting discussion to talk about the inner workings of a crack gang - o0o check out the HBO show, The Wire for all them nuggets!

[NewKidOnTheBlock]

Basic stats talks about how to analyze the numbers - not the context that those numbers fit.

I agree and that matters in the case of a study about search engine optimzation.

But there are so many things in real life for which you dont need an understanding of a certain field other than the field of real life .

For example one of the most common things I encounter is people assuming that b/c something occured (once) that that's the way it is.

Take a basketball game for example..people see a mediocre 3 point shooter make 4 out of 4 three pointers in a game (the only game with them which they saw) and they go "wooow! That guy is one of the best 3 point shooters Ive ever seen". Yet if they looked at the guy's numbers they'd see he's below average in a "big enough sample size".

And such lack of knowledge is often used and abused by people who make studies. Select the outcome you wish take a couple of insignificant samples and present the one that fits your picture to the crowd. Okay this might be exaggerated, but I guess you get what I mean?

It happens with so many everyday things for which you dont need to have a great understanding of the field.