Microsoft Quashes Search Relevancy Report

New
Search Engine From Microsoft Gets Cool Welcome from the Wall Street Journal
(paid reg. required) delves into the rumors we heard in August and confirms them
— that the latest Keynote findings on search relevancy have been suppressed by
Microsoft. The last we had were
back in
January. Why? Well, MSN Search is found to have slipped in those metrics.
For its part, Microsoft says it asked for the study not to be released because
it questions the methodology.

See? Further evidence it really is 1999 again. I used to get relevancy
figures from NPD back then in a scatter-shot approach. I’d know who was ranked
first, maybe third, but any search engine that didn’t think they were doing well
would quash releases of their own figures.

I covered this problem in my
In Search Of
The Relevancy Figure article in 2002:

Public release of this data is also important. Some companies that
contracted with eTesting Labs in the past refused to let the tests be made
public, if they did poorly. Similarly, the NPD Group used to do consumer
surveys, where the search engines‘ own users rated their relevancy. Those
search engines that did well often released their figures, while those that
did poorly kept quiet.

While it may be tempting to sit on bad news, if the search engines want us
to take seriously their claims of relevancy, then they have to agree to
release both the good and the bad. If a search engine does poorly, then that
poor performance should be an excuse to work harder.

I recently
dared
Google and Yahoo to report on relevancy and get past the size morass. So far, no
word. I can tell you that both companies have spent a serious amount of time
looking at the best way to spin the issue, far more than I suspect is being
spent on coming together to benchmark performance. Guess Microsoft won’t be
dared at all.

Honestly, if Microsoft and Yahoo execs can collaborate with Google for its
stealth Zeitgeist
strategic partner meeting, is it really too much to think they can’t come
together to benchmark relevancy? I know intimately the difficulties in measuring
such a subjective thing. But this is your CORE product, gang. This is what you
do — serve search results, and you have no way of proving to the world how good
you are. Really, screw knowing who is biggest — we don’t even know which of you
is most relevant!

It goes to the business model so crucially. More and more I hear about people
talking about Google’s search not being as good. Is it? What’s the proof of this
beyond ego searches? Traffic numbers staying up are one indication it might be
doing OK. But the more Google moves away from core search into areas like, hmm,
chat and wireless internet access, the more it becomes vulnerable to the “you
aren’t staying focused” accusations. In turn, those help fuel believe that
relevancy might be worse, even if it’s not.

In short, even the relevancy winner according to anecdotes and word-of-mouth,
Google, has a vested interest showing a relevancy fitness report to the public
that can be believed. That’s because anecdotes and word-of-mouth can easily turn
against it. Google, like all the major players, need to get us the benchmarks.

Meanwhile, I’ll say this. Microsoft may be one of Keynote’s biggest revenue
streams as the WSJ reports and thus has clout to help Keynote make a “business
decision” not to release the findings. But here’s another business decision to
consider. You’ve just wiped out any faith I have in your figures at all.

The next time Keynote trots them out — assuming there’s a better business
climate allowing that to happen — I’m going to be sitting here dubious if I
should even trust them.

Honestly, I can’t say enough how tired I am — and how tired everyone should
be — of these types of games. Google’s growing its index —
fact — but
won’t budge the home page number until it obviously feels it will be able to
wallop any counterclaims by Yahoo. So that number on the home page is
meaningless other than as a PR
hamburger count
stunt. Yahoo played the game as well, of course, deciding it was prudent to
finally publish its figures only when knew they’d have a count far larger than
Google could trot out. Microsoft’s playing the “don’t release relevancy figures”
game.

We did this once already in Web 1.0 or Search 1.0 or whatever you want to
call it. If we’re in Web 2.0 or Search 2.0, let’s see things advance. No more
games. Come together and go fund someone to properly study and benchmark how you
are doing, and publicly pledge you will live and die by those results. Don’t
roll back into the rut of last decade.

Aside from the relevancy quashing, the story also looks at how Microsoft
doesn’t seem to be gaining in the search wars despite the technological
investment in build its own search engine and backing it with marketing money.
It has been noticeable as when I did the various traffic numbers
last month
how MSN hasn’t really gained share. One interesting tidbit also is Microsoft
thinking of sharing ad revenue with computer makers to help lock in potential
search traffic.