Scraping the Engines - Other Choices?

[rustybrick]

Originally Posted by Marcia

Ok, so if that is the issue, what has it got to do with development of a tool being abandoned because of violating Google's TOS? Maybe that is what needs to be clarified for us. What has one got to do with the other?

Its about choices. The tool inspired the post. Google says no, dont scrap us. To be honest, I don't know why they would want to scrap Google. When I thought about it, there is no point to scrap Google or even use their API. Why? because it is a game.

Now Yahoo and the other engines don't have an API. So that tool, inspired the post about "choices". We don't have many, IMO - because the bridge has not been built with any substance.

[Marcia]

Many people, like Danny Sullivan, has done so much to try to improve the relationship.

Right - good point made there. But then again, others come along with plans to do things that will antagonize and cause deterioration of the relationship. Re-read GoogleGuy's forum post over there, it couldn't be clearer.

Synopsis:
1. Someone sets up a forum site and tries to develop a user base and traffic.
2. Someone then wants to monetize their forum site, which is perfectly normal and understandable.
3. Developing and offering a free tool has the usual effect of increasing inbound links and PageRank.
4. Selling text links brings revenue that's dependent on PageRank (see #3).
5. Selling text links is a way to monetize sites that many are taking advantage of - ergo the "sponsor."
6. The search engine with the clout of mucho "green power" waves a stick and says, like the dollar bill, "don't tread on me"
7. Said parties see the stick waving and back off.

So where's the connection and what's the problem?

[JasonD]

Hi Marcia,

I suppose my thoughts are biased as I try to speak about the general philosophies but I am involved with the building of the "specific tool"

I'll do my best to keep it on the philosophies from here on in

It has ALWAYS been a violation of Google's Terms of Usage to use automated tools, site scrapers, whatever to query their engine. And it is their site, so it's their privilege to limit anything they want however they want to. So how all of a sudden is anyone an innocent victim if they are not being allowed to?

I agree entirely with your comments and thoughts. You're right in saying that Google can do with their site and their TOS as they wish. It's theirs and theirs alone. If I or anyone else doesn't like it I can go elsewhere or alternatively I can ignore it and take the repercussions.

My view point is simple in essence. The reality is that whether there is a TOS in place or not these sites get scraped.

Notice I said sites, and not Google specifically. This is NOT a Google issue but a data issue.

Now, although I said this isn't site specific, and as I said above, Google were wise enough to understand the value of their data and came up with a solution. - The G API.

It made life easier, functional and legitimate to use the data they hold, but even with this API scraping still goes on and the reason is simple as to why that is.

The G API restricts what you can do with it (Certain searches don't work) and it limits the amount of searches you can undertake!

The other engines, as yet, do not even offer a restricted feed as G do so ultimately we are in the same position for all engines. If you want data then go get it from the search box!

I believe it doesn't matter in practice whether you automate or manually search as the analysis that is undertaken is pretty much the same. It just takes longer doing it manually compared to having code do it for you.

Because this is going on (although under the threat of breeches of service) and you, I, the search engines and other data owners knowing about it (although not talking about it) then wouldn't it be better for the engines to work with the people undertaking the analysis rather than not?

Surely open and honest discussion can deliver nothing but a win for all parties involved and simply ignoring the very real costs that come to bear on an engine caused by illegitimate scraping being pretty much eliminated whilst giving the community of "data gatherers" a manner to get their data. It surely can't be right that unofficial data gathering by hand is OK but unofficial data gathering by code is not.

The costs are the same either way and if the engines don't want the data to be available then they can turn off the SERPs. We know that SERPs will be available for a long time to come and whilst that is the case working out a way that reduces costs, inconvenience, hassle, management and time by allowing full access to the data in the same way a manual search does surely will stop the scraping being undertaken whilst delivering a manageable resource to the engines.

Ignoring the problem by sweeping it under the carpet can't be the way forward IMHO

[rustybrick]

So let's stop with the games.

[JasonD]

Originally Posted by rustybrick

So let's stop with the games.

Agreed. Do any engines want to chat with me as I'm more than happy to chat with you?

[rustybrick]

Just to be clear. I am talking about the games being played on both sides.

[JasonD]

Originally Posted by rustybrick

Just to be clear. I am talking about the games being played on both sides.

You're right mate. Whilst I sure don't speak for the community at large (this thread shows that not everyone agrees with me) I do believe that a lot of people in the SEO arena do.

I hope that my invitation for speaking with the engines is seen as an honest and open position to try and find an answer and not taking as game playing.

[Chris Boggs]

I have heard the opinion that many SEO's do not like tools, for the main reason that it offers potential clients the ability to perform rudementary SEO on their own. It seems like this thread has taken a little bit of a turn that way...I will start a poll to see what professional SEO's in here really think about SEO tools in general. See this thread.

[Everyman]

What is the purpose of the scraping?

1) To draw traffic to the site that sponsors the tool.
2) To sell ads that make money for the site that sponsors the tool.
3) To help SEOs generate more effective spam.
4) To inform the public about the functioning of search engines.

If your purpose is to inform the public, I say go ahead and scrape. Google's terms of service are nothing more than their recommendation of how we should behave so that they can make additional billions off of our content.

If Google doesn't like your scraping, they can block you. Or they can install IP throttling like Yahoo has, so that it will be all automatic. If you aren't willing to risk a Google block, then don't scrape. If you think you can get around Google's blocking or Yahoo's throttling, then what's the problem?

Are you worried that Google will penalize your sites? Heck, they'll disappear them for no reason at all, whenever they install some new broken algorithm, and you'll have no recourse but to sit around for many months and hope things improve.

Maybe you think Google actually deserves a $50 billion market cap?

Well then, learn how to googlestep because they've already brainwashed you.

[Mikkel deMib Svendsen]

Why does this remind me so uch of the Search King case ...?

I tell you why!
This tool could have been done, and similar tools ARE in fact being operated as we speak but if you provoke the engines like this by doing it all in public you are asking for it, in my opinion.

If you want to spam, do tricky things or go against Googles (or anyone elses) TOS then fine with me but please, if you do it in public you should know whats coming to you.

[Chris_D]

Isn't this the clause that prevents these kinds of tools?

Personal Use Only
The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance. Please contact us for more information.

http://www.google.com/terms_of_service.html

So Adsense, Adwords, API use etc are covered under the "agreement with Google to do so in advance"

[JasonD]

Please contact us for more information.

Which is why I am trying to open debate and discussion with the engines.

The engines say, "Speak to us" and that's what I am trying to do.

[Marcia]

>>Isn't this the clause that prevents these kinds of tools?

Personal Use Only
The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance. Please contact us for more information.

Looks like exactly the clause that's meant to prevent it, the telling element being "If you want to make commercial use of the Google Services."

The engines say, "Speak to us"

I'd interpret that to mean "Speak to us about pricing."

There are, in fact, sites that use Google's data - but I think it's safe to assume that they pay for using it. Like Infospace, who tell here which engines they get results from

http://www.infospaceinc.com/search/consumers.php

I imagine it was the Silver/Gold that it may have pertained to (at the time when it was still available), which does talk about credit card payments.

http://www.google.com/faq_silvergold.html

Other uses and the terms are probably all covered in the other sections of their Business Solutions/Development section.

http://www.google.com/services/index.html