View Full Version : Search Quality Panel?

08-18-2005, 10:33 PM
The American Heritage Dictionary employs the following Usage Panel, which contributes to the effort of judging marginal usages of words like "disingenuous" (94% of the panel agree with the dominant definition, for example).


The point being, in talking about language, we can go to the extreme of "definitions are in the eye of the beholder" or the extreme of "the dictionary (or the powers that be, or the church, etc.) is always right."

When judging the relevancy of search engines' results on common queries, wouldn't it make sense to form a similar panel, as a body that would have no power but could lend weight and impartiality to measurements of how the search engines are doing?

Marketing Guy
08-19-2005, 07:09 AM
Good idea in principle - I think the major SE's may have embraced the concept to a certain level already (authority hubs, eval.google.com, etc) - the difficulty would be choosing the panel.

Choose government representatives and you get people not really familiar with the issues of the industry. Choose SEO representatives you get people with bias and commerical interests. Choose SE representatives you get people with some bias (and perhaps an unwillingness to share opinions due to intellectual property issues - ie how "they" view relevancy).

Plus a fundamental difference between the dictionary analogy and the SEO industry is that the scope and impact is far less. Slightly altering the meaning of a word in a dicitionary has little impact. On the other hand, altering the relevancy of a SERP would impact lots of sites and potentially cascade onto other SERPs (through synonym values, etc).

Even now, the average SEO will pick apart any small point or comment (and usually come up with some half baked theory about it) - imagine what it would be like if they had a small independent body they could target their agression at (justifyably or not)? Would make the old update threads on WmW look like a day out in the park. ;)

*OMFG!!1!! {independent body} are targeting me!!! Why doesn't Google do anything about this????? I used to rank number 1 for "blue widgets covered in cheese for sale in Boston loans mortgages finance" but now I don't because they changed *the relevancy*!!! This isn't fair!!!!*


Perhaps setting up industry specific panels may be effective? At least then the members would be qualified to comment on the subject area (and people would be less likely to moan about it).

But then there will always be grey areas - where does information stop being relevant and shopping sites become more relevant for *that* query? Are people researching or buying?

An example in one of my subject areas. The top "information" keyword is dominated by multinational brands, national newspapers and information sites (like mine).

The brands offer information as a subsidiary to their service. The newspapers mostly offering information primarily and the service (to a lesser extent) as a secondary service. The information sites offer the information only.

Which is most relevant for that query? Perhaps the current mix of all 3 types is the most relevant result?

A business may argue that the companies offering the services are the most relevant because ultimately users will end up using their (free) service). Academics are perhaps more likely to favour newspapers and information sites. Independents such as myself may argue that pure relevance is the information sites - there are other terms for the other types of sites to rank on.

It's a very grey area (recruitment). Although people search for advice which would suggest favouring information sites, they almost always certainly subsequently use the services of a business (job board, etc).

Maybe the source isn't the issue - the quality of the information would dictate relevance - in which case, the panel would have to be knowledgable about the subject area?

Random thoughts - just ignore if they dont make sense! ;)


08-19-2005, 11:59 PM
A couple of points.

The model of the dictionary panel is pretty interesting because it draws upon many who make words and usage their passion, but who don't all share the same background. You had cultural leading lights like Tony Randall, government people, professors, and many authors from different walks of life.

The search panel would have to be leading lights from a wide cross-section of society, but they should have some professional competence in the area of research and relevancy. As long as the panel is large and eclectic and not particularly tied to any ruling body, then it becomes a sort of independent body.

Hey, they can do it on American Idol with three people. :)

As for there not being much impact of changing a dictionary definition, I would have to disagree! Language has shaped whole societies. Whether dictionaries have had much clout there is another question. In any case, the usage panels seem to have a sort of non-binding, informative power. It is of background interest to know what the state of thinking is about the dominant and contested meanings of words.

SE's use their own evaluators, certainly. But an independent panel with a different makeup, with its actions made public, could contribute to public discourse about search quality. That's never going to be a mainstream topic, but it will be of interest to about as many folks as are interested in semiotics and such. An important part of how society generates meaning... of immense interest to educated folks, and having a huge impact on everyone in one way or another.

There is some support in the literature about the "quality of democracy" that suggests independent panels in professional fields can contribute to enlightened discourse that can lead to social improvement, whereas if such panels and bodies are exclusively the province of private industries, they contribute mainly to industries' lobby efforts connected with their own self-interest. So, seemingly ineffectual groups that are carefully designed by selecting a cross-section of disinterested experts and informed non-experts are seen by some as a vital defense against the growth of one-sided, corporate discourse, even if they don't hold formal power.

One hurdle might be to find some scientists out there who aren't employed by Google, Yahoo, or Microsoft. :)