Secrets of http://eval.google.com revealed?

[JohnW]

An interesting article. Enjoy.


http://www.searchbistro.com/index.php?/archives/19-Google-Secret-Lab,-Prelude.html

[Nacho]

Outstanding find John!!! I can already see this story light up all search industry media like a Christmas tree.

[incrediblehelp]

very cool, I can't wait to see the next post.

[St0n3y]

Is Google really worried about the quality fo results for "free orgy pics", "free ametuer pictures"? You would think there would be other higher priorities.

[unreviewed]

Henk van Ess, is breaking AdSense TOS, asking people to click Google ads.


"Click on the Google Ads if you want to pay my host. Oh well. Never mind. Got a good salary."

[AussieWebmaster]

Henk van Ess, is breaking AdSense TOS, asking people to click Google ads.


"Click on the Google Ads if you want to pay my host. Oh well. Never mind. Got a good salary."
Let's give him credit for uncovering a huge action on Google's part.... and let him beta test if the request for clicks actually works!!!

[Henk van Ess]

This posting teaches me that Dutch satire is not universal. I removed the remark.

Henk

PS The part of the 'good salary' was totally true.

[randfish]

Henk,

Glad to hear it and great work! Very impressive work; thank you for sharing with all of us. Can you let us know if you will be offering more information on the subject in the near future?

[Henk van Ess]

Some new details are posted just now:
http://www.searchbistro.com/index.php?/archives/23-Googles-Human-Quality-Evaluation.html

Will try to post new info on Friday, Saturday and Sunday.

[dannysullivan]

Henk, the material is very, very interesting to read :)

How many more parts to come? And can you clarify. Have you seen anything that suggests that Google is actually changing results directly in response to the ratings?

In other words, I've seen some people suggest that Google is somehow using these human reviewers to change the results. IE, if they found a bad site, it sounds like Google might immediately pull that site. A good site? That site might be boosted.

In contrast, my read of it is like this. Find a bad site? Google might down the line penalize that site, but not immediately, when you push the button. And that type of move doesn't shock me at all. They already have spam reporting and have already done "human intervention" to remove sites that way.

Find a good site? So far, I haven't seen anything to suggest they might immediately rig the results for a particular query to make that come up. So again, no human intervention to make something rise to the top.

Google will go on and one about not having human intervention in their results, and I've written in the past that's not entirely correct. They mean they won't set a particular order for any query -- and to date, that's actually seemed to be the case. In contrast, both MSN and Yahoo in the past might have preordered certain results. But they have long intervened in terms of pulling sites out of the index or applying hand penalties that impact the site on a range of terms, as many know well.

The most intriguing part was the brief mention you made of the whitelist. I want to hear more about that. What is this white list? Who is on it? How does Google explain it? Did they make it so the reviewers would leave certain sites along. You quoted:

Here is a non-exhaustive "white list" of the sites whose pages are not to be rated as Offensive (nor as Erroneous):

Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com;

What that all you found? It's amazing that they'd ask people to review results but then say, "Don't worry about these sites."

[Bernard]

YOU WORK AS GOOGLE RATER + Wanna earn some money?

I bid € 2.000,- for you, paid in cash for rating my site's pages
in a positive way. Get in contact with me over ....

lol... DMOZ editors aren't the (supposedly) only game in town anymore...

[I, Brian]

Here is a non-exhaustive "white list" of the sites whose pages are not to be rated as Offensive (nor as Erroneous):

Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com;
Well, these are obviously *very* quality sites, and it is surely comforting to know that Google's objectiveness is not suggested as being for sale.

[NFFC]

> Well, these are obviously *very* quality sites

I disagree.

Imho they are mainly just dupe listings of merchant feeds, little added value and clutter the SERP's with multiple listings for the same product from the same supplier. From a users POV I think they suck.

I'm very surprised they and there ilk would make a "whitelist", I struggle to see the difference between them and independent aff sites, apart from the obvious.

[Henk van Ess]

Will try to do a special section about Google's spam policy on Search Bistro (http://www.searchbistro.com) with all the details I know.

[incrediblehelp]

>From a users POV I think they suck.

Well this is obviously only your opinion considering most of these shopping search sites are very popular to end users and are considered by ebay, yahoo as great investments.

[Bernard]

I disagree.

Last edited by I, Brian : Today at 10:55 AM. Reason: Just playing with different methods of delivering sarcasm.

He shoots.. He scores!

[I, Brian]

> Well, these are obviously *very* quality sites

I disagree.

Imho they are mainly just dupe listings of merchant feeds, little added value and clutter the SERP's with multiple listings for the same product from the same supplier. From a users POV I think they suck.


Next time I'll use the special [sarcasm] tag to be a lot more obvious. :)

[Henk van Ess]

Henk, the material is very, very interesting to read. How many more parts to come?

Posted two updates today. One is about Google's CommQuest - the mediator for raters when they disagree with each other and Google's full guidelines of random-query evaluation. Will publish at least two more entries: Google's spam policy and something about EWOQ, a rating system of Google

Only a fraction of the 8.058.044.651 URL's of Google are rated. What are the criteria for rating a source? I discovered no solid criteria yet. (So now and then hundreds of sources are scheduled to be rated, but the international agents don't have time to review them - the URL's just vanish and are replaced by new ones).

[mykel79]

Good to hear you'll be posting the spam policy. I'd love to read it, as I'm sure many here would.

And of course - great work!

[mykel79]

Anyone read the spam guide yet?
I don't understand the part about redirects.
Why is <snip> a 'sneaky redirect' and <snip> not?

I'd be grateful if someone could explain the logic to me :)

[Henk van Ess]

I think you just asked a rhetorical question.

[mykel79]

Of course to me it's evident that google's 'unbiased' opinion is for sale, but maybe some people have different views and I'm just being paranoid. Then again, maybe not;)

[NFFC]

I must admit to finding the "revelations" fascinating although in truth there is nothing in there [so far!] that isn't well known.

What really strikes me though is that nothing there really damages Google, maybe this will be the prod they need to make their webmaster guidelines more helpful, all they need to do is publish their internal eval procedures. They could swap out the real sites with some mocked up examples and be good to go. Win, win again.

[AussieWebmaster]

I must admit to finding the "revelations" fascinating although in truth there is nothing in there [so far!] that isn't well known.

What really strikes me though is that nothing there really damages Google, maybe this will be the prod they need to make their webmaster guidelines more helpful, all they need to do is publish their internal eval procedures. They could swap out the real sites with some mocked up examples and be good to go. Win, win again.
I agree but it is good to have what was solid speculation in some areas confirmed.

[mykel79]

I don't fully agree. To find it a little damaging to Google that with most rules, they first preach them to raters and then go and give whitelists of sites that clearly violate them, but 'should not be rated offensive/erroneous'.

In fact, I'm sure at least some raters are confused by these and are (or pretend to be) naive enough to ask for an explanation. I'm very curious how Google goes about doing that.

As for the general rules I also think they should just post them in the general webmaster guidelines. I'm looking through all my sites and have to admit that a lot of them will have to be improved. While all this was speculation I could convince myself that Google will in fact consider them valuable. Now I just have to go and make better sites if they're to withstand scrutiny by a human tester.

[seobook]

The whitelists might be partly due to financial relationships, but if Google has already decided that certain sites are above board then does it make sense for them to pay third party people to rate them differently?

[mykel79]

Exactly! I would think it would just confuse the raters.

Then again the raters are to decide if serp 1 or serp 2 is better. If some URLs were filtered and others inserted instead of them, then it would skew the results.

Edit: A thought I just had: I wonder if human raters are an element in Adsense smart pricing.

[msgraph]

>>Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com;

I remember in Jan. 2004, all of those sites took up almost the whole top 20 results for a huge amount of searches.

[Bernard]

Re: 100% Frame (from the spam guide Henk referenced) - I wrote a freeware affiliate link protector (http://www.measuring-up.com/html-tools/profit-protector.html) which uses a 100% frame to render an affiliate site. But I developed the tool specifically for assisting people to prevent affiliate ID hijacking. It had nothing to do with manipulating Google. In fact, the links work through Javascript, so the affiliate link frame pages shouldn't even appear in Google's index. This document/guide doesn't mention that there are legitimate uses for a 100% frame.

[incrediblehelp]

>>Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com;

I remember in Jan. 2004, all of those sites took up almost the whole top 20 results for a huge amount of searches.

I don't think these websites are getting preference in promotion, rather preference in trust and not being demoted. Websites like kelkoo.co.uk can still be penalized or demoted through the regular Google search algo we all try to figure out. 302 redirects, scrapper websites can still hurt the big guys just as much as the little guys.

[palms]

<<Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com>>

How do we know that these sites aren't monitored at a higher supervisory level than the "rater" level? Their size and influence with the public would certainly warrant it.

[incrediblehelp]

Sure they might be monitored at a higher level, but believe me that these websites still have to "please" the Google algo just as much as we do.

[AussieWebmaster]

I imagine that while the raters may see sites removed through their input I think ultimately the order of what is left will be back to the algo with a few of the ratings used in the mix... how it all works in the end should be obvious by Christmas!!!

[Henk van Ess]

Google confirms: Eval.google.com Exists But Stop Publishing Details Please (http://www.searchbistro.com/index.php?/archives/30-Google-confirms-Eval.google.com-Exists-But-Stop-Publishing-Details-Please.html)

[I, Brian]

>>Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com;

I remember in Jan. 2004, all of those sites took up almost the whole top 20 results for a huge amount of searches. A number of them are still in there. Absolutely couldn't possibly ever be a Trusted Feed. No, not Google.

[Dominic]

Why have a whitelist if the data input by human editors doesn't interact in real time with search results?

[incrediblehelp]

I am still trying to understand why everyone feels it is wrong for Google to get real humans to sample and recommend changes to their results. Doesn't this in the end help make their results better for everyone? If they do or don't use these "recommendations" in the search results, aren't they in the end trying to make their index much better to the end user?

Of course if these "recommendations" were actually used in the search results I could definitely see an issue with honesty or integrity of the each result down the road, similar to what has happened to DMOZ.

[AussieWebmaster]

I think if the human eyes are used to filter out the obvious spam is in itself a great effort. The amount of it would keep an army busy for a long time. To have them evaluating sites that meet the basics is a waste and open to too much potential for problems.

Train them to answer the numerous problems that regualr webmasters find their sites having without explanations.

[St0n3y]

OK, I'm confused. Are they revieing the sites themselves or the relevance of the sites that come up for any particular search query? I think there is a big difference between the two. Any site may be determined to be a great site, but if it pushes out other more relevant sites for any particular query then that shows the algo needs adjustment, without applying anything negative to the site itself. Any site being link bombed is a good example.

[PhilC]

Am I missing something or what? I don't see anything extraordinary about it. Why wouldn't a search engine want input concerning their serps in non-English languages, which is was the jobs ad was for, or even in the English language results? What's so odd about it?

I can't say that I've read every word (the searchbistro blog's navigation leaves a lot to be desired), but I don't see anything sinister in the parts that I've read. There's no suggestion anywhere that the outside people are able to do anything other than express opinions about the search results. *Everyone* is able to do that - there's a link at the foot of the serps themselves for that purpose. Of course, the paid people's opinions will carry weight, because Google is paying for their opinions, but the idea of outsiders evaluating the results is nothing new. The only new thing that I see is that Google has formalised it, and is paying for the opinions. It seems perfectly fine to me.

Or have I really missed something somewhere?

[AussieWebmaster]

Am I missing something or what? I don't see anything extraordinary about it. Why wouldn't a search engine want input concerning their serps in non-English languages, which is was the jobs ad was for, or even in the English language results? What's so odd about it?

I can't say that I've read every word (the searchbistro blog's navigation leaves a lot to be desired), but I don't see anything sinister in the parts that I've read. There's no suggestion anywhere that the outside people are able to do anything other than express opinions about the search results. *Everyone* is able to do that - there's a link at the foot of the serps themselves for that purpose. Of course, the paid people's opinions will carry weight, because Google is paying for their opinions, but the idea of outsiders evaluating the results is nothing new. The only new thing that I see is that Google has formalised it, and is paying for the opinions. It seems perfectly fine to me.

Or have I really missed something somewhere?
The information of the rating scheme and the qualification of spam gives some informal insights into the Google methods.

[PhilC]

And that's what the fuss has been about? Google pays a group of people to tell them of spam in the results? It all seems very reasonable to me.

But I don't see it as Google's own span cops, because there seems to be much more to it than that. The group appears to be evalutaing the results, and not just hitting on spam. In fact my guess is that any spam that's found is a side benefit, and not the primary objective. If the primary objective was spam, they wouldn't have been advertising for people who spoke certain languages fluently - they would have been looking for all languages.