Discuss the 5th Annual Search Engine Watch Awards

[rustybrick]

SEW team released the 5th Annual Search Engine Watch Awards today. Wanted to know what your thoughts were on this.

Here are some of mine, to get the ball rolling.

Outstanding Search Service <--- overall I am happy with these results, Yahoo deserves it.
Winner: Yahoo
Second Place: Google
Honorable Mention: Ask Jeeves


Best News Search Engine <-- surprised Google got the win
Winner: Google News
Second Place: Yahoo News
Honorable Mention: MSN Newsbot & Topix

Best Blog/Feed Search Engine <-- Yup, bloglines rocks, wonder why My Yahoo wasn't in there as well.
Winner: Bloglines
Second Place: Feedster
Honorable Mention: Technorati

Best Image Search Engine <-- IMO Ask Jeeves deserves the win for this, oh well, just an honorable mention.
Winner: Yahoo Images
Second Place: Google Images
Honorable Mention: Ask Jeeves Pictures & Picsearch

Best US Shopping Search Engine <-- I rarely use Froogle and I am an avid online shopper (i use bizrate, pricegrabber, shopping.com and sometimes yahoo shopping) I have tried Froogle, never happy.
Winner: Google's Froogle
Second Place: Yahoo Shopping & Shopping.com
Honorable Mention: PriceGrabber & Shopzilla

Most SEO/Webmaster Friendly Search Provider <-- MSN is by far the easiest to manipulate (I am told).
Winner: Google
Second Place: Yahoo
Honorable Mention: MSN Search

Best Search Ads Provider <-- sounds good to me, wonder what type of impact MSN will make next year.
Winners: Google AdWords
Second Place: Yahoo/Overture

Best Specialty Search Engine <--- Local Local Local
Honorable Mention: Google Local, Yahoo Local, Google Scholar, Scirus, Citeseer, Librarians' Index To The Internet

[I, Brian]

From "Outstanding Search Service"

[Yahoo!] gained only 17 percent of the 706 valid votes cast in this category. Nevertheless, we felt the achievements it has made warranted overriding the popular vote.

Have I misunderstood something in thinking that the voting process was a complete waste of time, because the winners are already pre-selected?

Search Engine Watch editor Danny Sullivan and associate editor Chris Sherman made the final decisions about award winners. These were influenced by reader votes, though the final decisions over winners wasn't always the same as the voting.

I'm sort of coming away with the impression that you asked us members for our time - but if you didn't agree with our opinions then you'd just bin it and write your own awards anyway.

So - what's the point of a poll??

[NFFC]

>Have I misunderstood something in thinking that the voting process was a complete waste of time, because the winners are already pre-selected?

I'm sure the votes counted and influenced the final results of all the catagories, I think it could be a little clearer that the votes are just suggestions but....

Seems the SEW team put their neck on the line each year, its what *they* think is the best whatever that wins. I like that, just because 99% of people vote the same way doesn't mean they are right, no?

[dannysullivan]

The time is appreciated, Brian. And we say that at the end of the awards page. But we also said right on the initial voting page that votes are something we consider, that they do help us in making the final decision, but we don't say that we always go with the votes.

It says that right at the top of the final winners page, as well, in the "How The Winners Were Selected" section:

Search Engine Watch editor Danny Sullivan and associate editor Chris Sherman made the final decisions about award winners. These were influenced by reader votes, though the final decisions over winners wasn't always the same as the voting. More details about how decisions were made are described in each category below.

Believe me, I don't want anyone to feel they've wasted their time -- nor do I think they have. Below, I'll list all the areas where we went with exactly what the readers wanted. But first, let me recap the overrides.

Outstanding Search Service: Yahoo

This is explained on the awards page, but the short answer is that we didn't think Google could be the best "overall" search service when it self-admittedly had let Google Images be stale for around 8 months of 2004. That's why it didn't get to win. If it hadn't been for Google Images, it would have tied with Yahoo. As for Yahoo, we thought it was good enough that it deserved to win regardless of the popular vote. We've done this in the past, recognized a service even if the popular vote does not because sometimes Chris and I simply think it deserves that attention. But the vote are definitely considered.

Meta Search: Jux2

Another case where we overrode the popular vote. Why? Few know of this service, so few are likely to have voted for it -- but it deserves plenty of attention and recognition. Check it out.

The popular vote would have made Dogpile first. We explained why we went against the popular vote. We just think that people using a meta search engine ought to be easily able to pick and choose exactly what they want. Dogpile doesn't allow this. The heavy mixing of paid and unpaid together was also a drawback. Again, the awards page explains more.

Images: Yahoo

Again, while we recognize the popularity of Google Images in the voting, the fact that for something like eight months it was out of date simply didn't make it possible to us ultimately to let it win.

That's three overrides in all. We also gave Mamma and Shopzilla honorable mentions despite being "fourth places" in both cases because we thought they were close enough. Now what did we go with exactly the readers want?

Best News Search Engine
Best Blog/Feed Search Engine
Best Image Search Engine
Best Shopping Search Engine
Most SEO/Webmaster Friendly Search Provider
Best Search Ads Provider

And as much as possible, the same thing for Best Search Feature and Best Specialty Search Engine. Those are hard, because they are completely open ended.

So I'll I can say is that the poll results are definitely very important to us -- that's also why we list the exact result for everyone on that page. But from the very beginning, we've felt the editors should ultimately make the final decision, and we've always been clear about that right on the voting form itself, as well as with the write-ups.

You never please everyone, of course. Look at rustybrick. He's surprised Google News won, thought Ask should be there for image search, didn't think Froogle should win, loves that Yahoo won. In some cases, what he didn't like is a result of reader votes. In some cases, what he likes is what we overrode. And other options like that, as well.

Anyway, hope that helps. Plus, I'd definitely love to hear feedback on what anyone would like to see happen to improve or change the awards going forward. Should the editors completely go with reader votes? New categories?

[I, Brian]

It simply seems counter intuitive to invite a poll and then simply discard the results of it. Well, outside of the Ukraine, at least.

Perhaps you could consider simply making the awards yourselves on the basis of your own perceptions - and if you still wish to invite a poll, publish the results as an additional popular vote section?

Maybe the popular vote isn't going to offer a proper perspective on the leading edge of the search engine industry, but it's still a perspective.

At least then we can have some interesting metrics on user perceptions, though it risks separating SEW opinion from popular opinion in a perhaps more overt way.

However it would be interesting to see if popular opinion follows SEW perceptions over time.

Overall, it just seems completely pointless to offer a poll if the results are routinely discarded - and IMO it works against the supplier-consumer relationship.

[FeldBum]

Aww...Clusty didn't win. I love Clusty!

[nemnem]

Originally Posted by I, Brian

It simply seems counter intuitive to invite a poll and then simply discard the results of it. Well, outside of the Ukraine, at least.

Perhaps you could consider simply making the awards yourselves on the basis of your own perceptions - and if you still wish to invite a poll, publish the results as an additional popular vote section?

Maybe the popular vote isn't going to offer a proper perspective on the leading edge of the search engine industry, but it's still a perspective.

At least then we can have some interesting metrics on user perceptions, though it risks separating SEW opinion from popular opinion in a perhaps more overt way.

However it would be interesting to see if popular opinion follows SEW perceptions over time.

Overall, it just seems completely pointless to offer a poll if the results are routinely discarded - and IMO it works against the supplier-consumer relationship.

Maybe you guys should use the electoral college system?

[Chris Sherman]

Originally Posted by I, Brian

Overall, it just seems completely pointless to offer a poll if the results are routinely discarded - and IMO it works against the supplier-consumer relationship.

We didn't discard the results of the popular vote - far from it. The voting form not only allowed a vote, it allowed users to make comments. We looked long and hard at those comments, as well as the raw scores. Very valuable input, believe me.

Chris

[dannysullivan]

Perhaps you could consider simply making the awards yourselves on the basis of your own perceptions - and if you still wish to invite a poll, publish the results as an additional popular vote section?

That's basically what we've done. Go back and look. Every one listed is shown with how they polled -- and everyone who made it polled in the top three in voting is there in some way, usually in the order they polled. If you want the popular perception, it's all there. Honestly, we've been pretty much at pains to ensure you know exactly how people voted in each case.

We could I suppose do a further break out and do little tables for each category. Better, I think you'd like to have a "People's Choice" award. As I said, in most cases, that's what you've already got. But if we do an override, it could be that we also give a People's Choice winning award somehow, as well.

Overall, it just seems completely pointless to offer a poll if the results are routinely discarded - and IMO it works against the supplier-consumer relationship.

I can only say what we said before. We didn't routinely discard anything. In most cases, we went with exactly how the polling went. When we did an override, we still didn't discard the voting in that we just said nah, forget that. We weighed heavily up the popular vote and especially comments that we got. We want the poll because we want to know what our readers think, and that's a way for them to lobby us as to what they think should get the final awards. But again, as I said earlier, we always have made it really clear that we'd make the final decision.

This is the first year anyone's been upset about this, honestly. But the feedback is good -- it just may be that the awards have matured to the point that we need to do something different. In fact, rather than a People's Choice, it might be that we give special editor wins in addition to the awards chosen by readers.

[AussieWebmaster]

Okay you read my comments earlier but I actually thinnk you and Chris should make the decisions - using our stuff as input.... and I further admire publishing the contrary numbers when we would be in the dark if you didn't.

Great job and congrats on doing this for 5 years.... hope to enjoy reading and "discussing" the awards for years to come!!!!

[Nacho]

Originally Posted by AussieWebmaster

Okay you read my comments earlier but I actually thinnk you and Chris should make the decisions - using our stuff as input.... and I further admire publishing the contrary numbers when we would be in the dark if you didn't.

Great job and congrats on doing this for 5 years.... hope to enjoy reading and "discussing" the awards for years to come!!!!

I second that!

Danny and Chris, thank you for letting us Members influence in your decisions by taking so much time to do it they way you have.

Awesome Job!

BTW! I was very shocked to see Google as #2 for outsanding search service, but after reading why I very much agree and support the editors on picking Yahoo! Yah, why not? Yahoo! Search does deserve it. Congratulations to the Tim(s) and the entire search crew for getting #1. Too bad they lost Jon Glick though.

[krisval]

Prediction: 2005 Winner of "Outstanding Search Service" MSN

I love that Yahoo is getting due respect. I think so many people are hooked on Google that they haven't given Yahoo a "new" try.

[AussieWebmaster]

Originally Posted by krisval

Prediction: 2005 Winner of "Outstanding Search Service" MSN

I love that Yahoo is getting due respect. I think so many people are hooked on Google that they haven't given Yahoo a "new" try.

Not if the results stay the same as they are right now!!!

And Yahoo is nearly as bad... new look or not....

I wonder if anyone noticed here that NewsIsFree has a new and greatly improved search engine. Besides being able to search multi-lingual (over 20 languages) it also covers the most non-blog news sites

Check it out - comments welcome
Link to Advanced Search form


Pino

[Webvisitor]

I read the results of the voting and came away assuming everyone in the world knew about Jux2 except me. Never heard of it but thanks to the promo I do now.

I understand your explanation Danny but kindly disagree with the reasoning.
This is like the mayor of a city disregarding the voters choices for city council members and installing another in place because he or she is a good person that nobody knows yet but has uncommon potential.

It matters because some of the (existing) metasearch properties work pretty hard to earn reputation and goodwill (and notice) and have the user data to prove their value. Some newcomer walks on the scene IN BETA and they get to pop a prestigious award badge on their front page.

I like the way you punished Google for the images lag and I agree Yahoo is a superior search and news source.

[Qal]

Perhaps you should do just like Webby Awards.

SEW Winner & People's Choice Winner in each category.

[Qal]

It matters because some of the (existing) metasearch properties work pretty hard to earn reputation and goodwill (and notice) and have the user data to prove their value. Some newcomer walks on the scene IN BETA and they get to pop a prestigious award badge on their front page.

Well, thats certainly not true. Existing (old) metasearch properties can't expect to win the awards each year without improving/adding a function in the whole damn year. Did you notice atleast one (1) change in Dogpile and Mamma from 2004 upto 2005?

Jux2 is the best metasearch engine currently for sure.

[PhilC]

I can see valid reasoning on both sides here:-

On the one hand, the awards are just the opinions of 2 people (not much better than 1 person), which makes them almost meaningless, even though the people are highly creditable. When you get down to it, who cares what 1 or 2 people think? No discredit to Danny and Chris is meant, but it's true. E.g. only 17% of the popular vote for "Outstanding Search Service" seems quite low for it to be overruled.

On the other hand, the results can't be a poll of all-comers because it wouldn't work. It's too easy to rig, votes can be cast for a myriad of services, and 17% might actually be the top vote by a long way.

I didn't see the voting, so I don't know how it works, but how about this:-

A panel (Danny and Chris, for instance) selects nominations for each categeroy - perhaps from public suggestions as well as their own. Then votes are cast for the nominated services - a poll. Something like the Oscars and Baftas.

If that's the way it's already done, forget it and I'll go back to bed

[AussieWebmaster]

Originally Posted by Qal

Perhaps you should do just like Webby Awards.

SEW Winner & People's Choice Winner in each category.

I like that idea.... what do you think Danny???

[dannysullivan]

On the one hand, the awards are just the opinions of 2 people (not much better than 1 person), which makes them almost meaningless, even though the people are highly creditable.

Well, fair to say, we don't think they are meaningless just because we don't always go with the popular vote. I can only say again that we don't just completely dismiss the popular vote, either. It, if you will, is like a third editor sitting with us as we deliberate. It just doesn't always win, regardless of how strongly it speaks out.

Only 17% of the popular vote for "Outstanding Search Service" seems quite low for it to be overruled.

We've overruled others with high votes, with Dogpile being the most noticable in the past. We simply felt that the failure to identify paid listings from non-paid didn't mean it could win in our books, regardless of the overall popularity.

On the other hand, the results can't be a poll of all-comers because it wouldn't work. It's too easy to rig, votes can be cast for a myriad of services, and 17% might actually be the top vote by a long way.

Yes, Google has something like 3,000 employees. Not hard to say go vote for us, and that could wipe out other services. We do ask companies not to vote for themselves, and checking we've done doesn't show this as a major problem. In the past, before votes were tied to registered email addresses, it was a much bigger issues. We had to do a lot more looking to decide if we were just being spammed with votes. Hence our general hesitation in the past to simply hand things out based on popularity.

A panel (Danny and Chris, for instance) selects nominations for each categeroy - perhaps from public suggestions as well as their own. Then votes are cast for the nominated services - a poll. Something like the Oscars and Baftas.

We've done the opposite. Open nominations for anyone who is a paid SEW member; then final voting by anyone who is a newsletter subscriber, with us making the ultimate decision.

It matters because some of the (existing) metasearch properties work pretty hard to earn reputation and goodwill (and notice) and have the user data to prove their value.

Having a lot of people use you isn't necessarily indicative of quality, sorry. There's a delay factor, where people may continue using you until they realize things just don't seem as good as they were in the past. There's a quality factor. Lots eat at McDonald's. Heck, I do. Is it the best restaurant just because In-And-Out has fewer outlets and thus fewer likely to know it and vote for it?

Perhaps you should do just like Webby Awards.
SEW Winner & People's Choice Winner in each category.

It's very likely we'll do something like this, a People's Choice and then an Editor's Choice if we disagree with the popular pick, I think. It's a good idea, and one reason I'm glad for the feedback. Until this year, it's honestly not been an issue that anyone has raised other than maybe a service itself that was passed over. I had literally not one email from anyone in the four previous awards complain that we did an editor override. But we've never done that in the Outstanding Search Service category, so clearly that's hit some more people as odd. The dual awards as needed is an excellent idea.