The coming collapse of search

[Nagle]

The basic assumptions that the search engine industry uses for rating commercial sites are coming apart.

In the beginning we had search by keywords, back in the AltaVista era. Keyword spam killed that, and now the major search engines don't even look at anything the user can't see.

Then the main metric was incoming links, Google's big innovation. Then came link farms. In the last year, link farms became big business. One link farm company, Marchex (ticker symbol MCHX), went public. Marchex has several hundred thousand domains, of which, according to the Wall Street Journal, "lasvegasvacations.com" is a typical example. Somebody named Kevin Ham not only has 300,000 domains, he cut a deal with the government of Cameroon to get all the typos from people who type ".cm" instead of ".com". Try anything in ".cm" to see one.

The next big idea was to look for links in blogs. So then we had blog spam. Then phony blogs by the millions. Then phony but plausible content to fill the phony blogs.

Between link farms and phony blogs, links today are mostly spam. Search engines try to evaluate and filter, but the filters leak, and with so much junk linkspam out there, they have to. When the filters get adjusted, search engine positioning changes drastically, as most of the people here know. Nobody is happy about that. General links just aren't a reliable source of "value" information about commercial web sites any more. We're reaching the point where the only links worth anything are in "authoritative" sources, like real newspapers.

Using user recommendations seemed promising. But that, of course, can be manipulated. There's now a company in the business of manipulating them in bulk, with an automated tool: Collactive. Collactive just got US$2M in funding from Sequoia Capital. Want a high rating in Digg? They'll do it for you. Their slogan: "It's good to be popular". That's going to make links on "authoritative" sites which allow user input meaningless.

Google has tried to crank in other info, like how long a domain has been registered. That worked for about a year, while the bogus domains aged. Google even patented a scheme for rating sites by how much they spend on advertising. That was greeted with derision in the trade press.

The fundamental problem is that all the data sources used for ranking by the major search engines are now being spammed by organizations with substantial resources. As "search engine optimization" has changed from a garage shop business to a major enterprise, the quality of the data that drives search engines has declined substantially. It's going to get worse before it gets better.

[shorebreak]

The very notion that the search engine's organic ranking algorithms should be kept free of any human interference on the other side of the fence is, IMO, wrong. I'd argue that self-interested involvement on the part of companies has and will continue to lead to better search results than the search engines could achieve on their own. In the same way that the ppc auction + Quality Score process has led to better ads and better sites that more quickly and efficiently translate the searcher's intent into results, so it goes with businesses doing what they need to do move up in the SERPs.

Granted, it's not perfect by any means, but as we're seeing with Google's fairly successful attack on click arbitrage firms, progress is being made.

So in summary, I disagree 100% with what you're saying. SERP's untouched by anything other than the cold hand of the SE's algo's would be a far worse result set than what we're evolving into today.

-Shorebreak

[piffany]

I don't disagree with what you are saying per se, but interestingly it seems to me that search engines are getting better. I don't like Google mixing the news, blogs, images, videos, etc into one big list, but if you can get past that, their results for techie related stuff like code is getting even better. Ask.com has made some astonishing leaps forward for more pedestrian searches. Yahoo now seems to be competitive with Googles results if you can get past their cluttered front end. Further, its generally true that spam will always break new ground, but likewise, so will the technology that defeats it. Search engines have always contended with spam and they probably always will. My prediction, search will continue to be the front end to the web and it will continue to get better as it always has and (lot of and's, sorry for that)...and spammers will continue to challenge and drive the technology forward. Final note, any time someone says its the end of the world, they usually end up looking foolish, e.g., Bob Metcalf "Windows NT will shut down the open source movement", Bill Gates "There will never be a need for more than 16K of memory" (not really quotes but paraphrases.)

Cheers

[beu]

When does forum spam come into play? I'm saving this as an example!

[NewKidOnTheBlock]

I'm sort of wondering, what point you're trying to make. Sure people will try to spam every new ranking factor and in turn the search engines try to adapt.
I mean it's not like the SERPs are utterly useless. Actually, they've become harder to manipulate, I think.

Sure sometimes, there are sites in them that dont belong, there, but all in all it's working (even if theres room for improvement).

What's your suggestion? As any ranking factor has a chance of being spammed not use any ranking algorithm at all and just use a huge directory instead :confused:

[caugas]

I think this topic further strengthens the theory of running clean SEO (good content + good links). I am 100% positive that the SERP's especially Google will continue to reward sites that run clean and precise SEO programs.

I don't think anyone would argue that Google employees some of the brightest people on earth. For every action there is a reaction. For example Google provides guidelines, they are followed, you are rewarded.

We see this with SEO (clean SEO) and SEM (quality score). Personally I don't look for the quick next greatest SEO scheme, i.e. link farms, forum spam, blog spamming, because your quick rewards today could equate to damaging results later (when Google shifts their algo).

So what I am trying to say is don't worry about what everyone else is doing or how they are doing. If you are a good SEO, you know what needs to be done and in what fashion it needs to be done in.

Quick Schemes ='s possibly damage
Good SEO ='s great positioning

[NewKidOnTheBlock]

Now, I agree hehe.

Though, I believe there are still a couple of blackhat SEOs out there who are making a decent living on spamming the engines.

But that will probably become harder and Im more interested in SEO from a marketing perspective and thus think long term and believe in leveraging seo techniques, that wont piss off the search engines.

I think the most underrated aspect of white hat seo, that I never really hear anybody talk about is understanding exactly why something ranks and using that knowledge to find great opportunities/markets in the first place.

Of course, this only applies to people running their own sites, but it still amazes me, that in those "does whitehat seo work?" debates everybody only talks about promoting and pushing sites in a competitive field, without ever mentioning how they can use their knowledge of how the search engines work in order to find great fields. To me that is part of SEO, too.

[caugas]

Though, I believe there are still a couple of blackhat SEOs out there who are making a decent living on spamming the engines.



I agree, there are plenty of blackhat SEO's making money hand over fist (lots of dough). But addressing the topic. These schemes or practices (black hat) are in no way going to effect the search industry in hole. The search industry is gigantic now, and revenues and visitors are only growing a unbelievable paces.

My earlier point was that Google and the other SERP's will/should/better/attempt to make sure that they protect there investments (i.e.) the marketers that spend money with them. They (the SERP's) will do this by countering suspect activity, i.e. algo changes to off site tilted positioning. I hope this makes sense!

[cryptblade]

The problem with prognosticators of doom are that they NEVER get it wrong because they NEVER get it.

There will be no doom to search engines or searches. The end will be a transition into something new - IF that.

T.S. Elliott wrote, in the Tin Men, "this is the way the world ends...not with a bang, but a whimper". Granted it was a poem and all, but still, illustrative of reality. Rome ended with a whimper. The Third Reich ended with final sigh...only Japan ended with a bang...But I digress....

Don't forget search engines are now MEDIA companies - Google is a MEDIA company (stock: GOOG). Yahoo is and has been a MEDIA company (stock: YHOO).

IF the search engines are going to change, it will be to THEIR advantage - not because "users" are dictating to them.

Do you really think that Google's $500+ per stock price isn't helping them with infusion of cash to pursue things still not even dreamt of by anyone here?

The engines will find a way to, constantly, to combat spam. But spammers will actually need to keep up with the engines. The engines are diversifying and brining search into other media.

In a way, they are making SEARCH a commodity. How long before users get used to seeing a search engine bar for ... Google, Yahoo.. or someone else, on every site? (like a diner with a phone book).

We are getting to a point where search engine USE is almost synonymous with Internet use.

If search engine USE becomes such a commodity, then all debates about the end of search or search spam will be pointless.

Example: why are Americans so focused on "alternative fuels" for cars instead of focusing on more public transportation? After all, more public transportation is better - cheaper for the public, no car insurance needed...no fuel costs...no road rage. But the car culture INFRASTRUCTURE is enormous. Consumers of cars - big for the economy. Insurance - big for the economy. Driver regulation - good gov't tax-base & fee revenue (gas tax, licenses, etc.). Road construction. Gas sales. Etc. - We know cars suck. Too many of them on the road. Use too much gas. Too much expense between gas, insurance, etc. But we NEED it because in America, we can't get anywhere without a car.

Search is becoming part of the media consumption INFRASTRUCTURE where we NEED it for just about anything.

In a short time, the debate will be...how do we use search better or else we risk being lost. Not... should search get better or else die and someone else takes over.