Directory Purge, beginning of the end for Dmoz ?

[aquestionoftime]

With the recent apparrent 'purge' of directories, and especially with the % of those that involved dmoz clones.
Is it going to be too much of a risk as far as Google serps are concerned to download the ODP RDF dump and use parts of it to have an 'easy' directory on site for the benefit or your own personal website users ?

After this week, would any seo professional advise anyone thinking of using the data and using parts of the odp rdf dump, is a good thing/bad thing/kind of risky ?

And as such, and with the amount of webmasters who have had to delete their clones and 'directory' section off site, will this mean that the importance of dmoz will diminish also ?

Given that for a fair %, it's proved fatal, to the point of the whole website being dropped off the face of the earth, to have odp data as part of their website even in a small capacity ?

Is it going to be too risky to download and use the dump ? And if so, what does it mean to the relavance of dmoz to seo 6 months/a year from now ?

I know there are other factors involved, but specifically focusing on dmoz and what seo experts would advise, given the events of the last 2 weeks.

Ps grateful if this one wasn't locked also. Cheers.

[birdie]

How many times over the years have we heard "the beginning of the end for DMOZ"? - none of them came true and I see no reason why, because Google drops a couple of DMOZ clones, that this even comes close to spelling the "the beginning of the end for DMOZ".
Ps grateful if this one wasn't locked also I am not a moderator here, but IMHO experience threads only get deleted, edited, locked when someone chooses to break forum rules.

[aquestionoftime]

So if you were an seo professional, would you consider it risky or not ?

Best to stay on topic like you said

"I am not a moderator here, but IMHO experience threads only get deleted, edited, locked when someone chooses to break forum rules."

Thank you, I shall keep it in mind in future.

So, risky or not ?

[Marcia]

How many times over the years have we heard "the beginning of the end for DMOZ"? - none of them came true and I see no reason why, because Google drops a couple of DMOZ clones, that this even comes close to spelling the "the beginning of the end for DMOZ". One has nothing to do with the other. The clones raise duplicate content and index quality issues, while DMOZ itself serves an entirely different purpose for the IR community, aside from it's function as a resource.

[cbp]

So, risky or not ?
I am not sure I can answer - I just do not see why anyone would want a DMOZ clone anyway????

The only good DMOZ clones are those that add value (eg Google --> rank by PR; Alexa --> Aexa info on site; Thumbshots.org --> Image of site) --- why would anyone want a DMOZ clone for the sake of it? Google et al should ban those that add no value --- if DMOZ has 600 000 categories, then if each clone has that many and there are a few 1000 clones ..... I can't be bothered doing the math, but that a very large number of pages in Google (and other SE's) index that is taking up storage space and computing power but not offering any value .... IMHO..

[PhilC]

Personally, I wouldn't add the DMOZ data to a site that matters just now. But I'm not so sure that directories are being targeted specifically. What's been happening recently could be due to duplicate content, as Marcia suggested, or it could be due to weeding out a certain type of page/site by profiling, and some directories fit the profile. If that's what it is, then it's unlikely that it's just directories that are being hit, imo.

Btw, I agree with Marcia that "one has nothing to do with the other". In the past, people have suggested the end for DMOZ for various reason - often sour grapes. But something is certainly happening now that's affecting some DMOZ clones, and just because DMOZ didn't die in the past doesn't mean that it won't die in the future. But I don't think that this marks the start of it. I think this is something different.

[projectphp]

I don't get why the title of this thread is "Directory Purge, beginning of the end for Dmoz ?" and yet the discussion is of DMOZ clones. the two are not the same. DMOZ doesn't need clones, and if every clone went all Sith on us and attacked themselves and all died, DMOZ wouldstill live on.

Thumbshots.org
Never seen that before and that is WAY COOL!!!!. (BTW: www.thumbshots.net is the directory)

After this week, would any seo professional advise anyone thinking of using the data and using parts of the odp rdf dump, is a good thing/bad thing/kind of risky ?
It always was a waste of time unless the reason was to add value.

While perhaps not spam per se, those are exactly the sorts of pages that any SE would not want to bother crawling. It costs them money to crawl all the DMOZ clones, so dumping them has always made sense IMHO.

If you like short term strategies, it still might make sense, but as fora longterm way to build a business, me thinks not!

[Marcia]

I don't get why the title of this thread is "Directory Purge, beginning of the end for Dmoz ?"Very good question.

Is there anyone who can come up with one halfway decent reason why purging the index of duplicate content would have one iota of an effect on Dmoz? Or why one has anything to do with the other? Can anyone come up with anything intelligent as to why the answer to "Directory purge, beginniing of the end for Dmoz?" isn't just flatly NO?

Or is this just designed to stir up the pot and see how many can be enticed to pick up a ladle and thow some more ingredients into the soup?

Methinks anything other than simply YES or NO is just an unnecessary waste of good celery and carrots.

[PhilC]

I don't like celery or carrots, so here ya go...

The idea of DMOZ is to create a directory that anyone can include in any site. It isn't being created for people to use the DMOZ directory itself. There's nothing stopping people from using it, but not many people actually do. If most sites that use the data in directory form run the risk of being dumped from Google, then perhaps a great many of the active editors will start to wonder why they are giving so much of their time freely to support the very few sites that are left - profit-making companies like like Google.

Suppose Google dumps all clones and the only site left using the data is Google - and they are *the* main users of the data. Why would anyone give their time freely to support a company that makes as much as they do? A small number of other big sites use the data, and they are also profit-making, so the same would apply.

That's one possible way that clones being dumped could cause a decline in the productivity within DMOZ to the extent that it could slowly grind to a halt. Another is the very reason that DMOZ exists - for people to use the data on their sites. If the biggest search engine (Google) effectively won't allow sites to use the data, because sites really do want to be listed in Google, then Google is interfering with the very reason that DMOZ exists, and, again, many active editors might wonder why they are continuing to produce the directory, etc.

[macdesign]

I actually don't resent Google using DMOZ, since I consider Google almost like a public service. I do not like the thousands of sites that copy DMOZ without adding any features and make money off my editing and reviewing sites, so I think this is a good move.

[Marcia]

First off, if you look at how many times in IR research papers and studies ODP is used as a base for seed sites - for accuracy of categorization and as a model for the taxonomy - the value of ODP to the entire search engine industry, and universities and research organizations is immeasurable. That in itself is a value that can't be judged by whether or not individual sites get in or not, or whether anyone likes it or uses it or not - or even whether it's all cloned.

It is what it is in itself - and has value for the industry as a whole that's beyond any individual need or use.

[PhilC]

But the reason that you, and all the editors, are editing in DMOZ is for the many sites that want to use the data, macdesign. Would you be so happy about it if Google was the only site that uses the data, and you are working for free for them? Perhaps you would, but if I were still an editor, I wouldn't like that at all. And I'd like it even less if it were Google who effectively prevented other sites from using the fruit of my freely given labours. But that's me.

I don't consider the comparitively tiny number of people of who may benefit from the uses you described merits the very large amount of freely given time that is continually being plowed into DMOZ, Marcia. And I can't agree that, for the uses you stated, DMOZ is a significant enough value to the industry as a whole, to merit the very large amount of free time that it takes to produce it.

The idea of adding value, that has been mentioned a couple of times, mystifies me. Adding value to what? DMOZ data can add value to a website, perhaps by reproducing the categories that are relevant to the site, but I don't see how any site can add value to the DMOZ data, and there's no reason why any site should add value to it. It's supposed to be pretty good as it stands.

Google doesn't add value to it, and it can hardly be said to add value to Google - they even took the link to their duplicate content clone directory off the front page.

[PhilC]

It's worth a reminder that it isn't just DMOZ data using directories and clones that have been dumped in recent days. Non-DMOZ data directories have also been dumped. It may be that Google has specifically gone after DMOZ clones in a seperate move to the move that dumped the other directories, but it could also be that they haven't gone after DMOZ clones at all, and that what they did just happened to catch some of them along with other directories.

Then again, I'm still inclined to think that they haven't even gone after directories specifically, but that a profile, or a dupe content run, caught some of them, along with other types of pages and sites.

[projectphp]

The idea of DMOZ is to create a directory that anyone can include in any site. It isn't being created for people to use the DMOZ directory itself. There's nothing stopping people from using it, but not many people actually do. If most sites that use the data in directory form run the risk of being dumped from Google, then perhaps a great many of the active editors will start to wonder why they are giving so much of their time freely to support the very few sites that are left - profit-making companies like like Google.
So, let me see if I can get this straight Phil. You are saying that this thread is about how DMOZ will die because Google is targetting directories. this makes sense because DMOZ exists to serve its RDF dump and create the RDF dump to help others, not to create DMOZ itself. Google's decision will mean the RDF dump is less used, and as such this will render the effort of DMOZ editors irrelevant and will signal an "exodus, movement of DMOZ editors" away from the directory because editors all exist to serve the RDF dump...

Not trying to be difficult, that is just so very different from what I understood of DMOZ editors motivations to be that I think I need a bit of time to consider whether that has any validity...

I don't consider the comparitively tiny number of people of who may benefit from the uses you described merits the very large amount of freely given time that is continually being plowed into DMOZ...
Do people in DMOZ think that way? Curious to see if that is part of their motivation in editting @ DMOZ...

DMOZ data can add value to a website, perhaps by reproducing the categories that are relevant to the site, but I don't see how any site can add value to the DMOZ data, and there's no reason why any site should add value to it. It's supposed to be pretty good as it stands.
To clarrify, adding value can be as simple as a better design. Or it can be as complicated as http://www.whois.sc/dmoz/forums.searchenginewatch.com, a site that takes DMOZ data and returns results based upon how many listings a site has, in reverse order.

IMHO, I don't see how being a full clone adds any value. Why not just link to DMOZ, or to the Google directory or another place that has DMOZ data in an improved format? Why take the time, effort and bandwidth hit to always be one RDF dump behind?

As an aside: has any existing site that has done as Phil mentioned here and added "... categories that are relevant..." to their site in addition to regular content been filtered? Or is it mostly DMOZ directories that are stand alone clones?

[PhilC]

I'm not suggesting that DMOZ will die. I was responding to Marcia's question about how dumping DMOZ clones could possibily cause DMOZ to die, and I suggested how it could happen. (I also suggested that Google isn't specifically dumping DMOZ clones, or even specifically dumping directories as such).

But if Google did dump all the clones except their own and maybe a handful of very big sites, they would certainly prevent DMOZ from fulfilling its purpose, and, if I were still an editor, I wouldn't work for Google for free - not when they are making billions - and especially since they were the ones who prevented the objective from continuing to be fulfilled. They would have virtually hijacked the DMOZ data, and I wouldn't help them.

Do people in DMOZ think that way? Curious to see if that is part of their motivation in editting @ DMOZ...I've no idea. I honestly don't know why I did it - I didn't last very long. donaldb has said he does it simply because he enjoys doing it, and for no other reason. But I imagine that most editors have an underlying sense that their work is for people to see and use - similar to an artist, photographer, music writer, etc., who do it for other people's benefit/pleasure.

I don't see what use a clone is either, but I think a niche section of DMOZ cats that are relevant to a particular site would be beneficial to a site and its visitors.

In the WMW thread it was directories that were being dumped, and not just DMOZ clones. I remember one chap's directory was a niche one, apparently the best in its field, only part of the site, and had been around quite a few years, but the whole site was dumped. Of course, there's nothing to say that it was the directory section that caused it. The fact that it has a directory section may have just been coincidence, but the timing of the dump was spot on with other directories that were dumped.

[macdesign]

But the reason that you, and all the editors, are editing in DMOZ is for the many sites that want to use the data, macdesign.Absolutely untrue as far as I'm concerned. I would just as soon that at least 80% of those clones disappeared, they degrade the value of ODP data by cluttering it with off topic paid listings with terrible descriptions or by copying it as is with no added value. If they all disappeared, I would keep on editing - being just as happy; and even if Google dropped DMOZ, I'd keep on editing.

Apart from anything else, I'm one of those [and I admit it might be a small number] who used DMOZ as as a resource before I became an editor. I had no idea there were DMOZ clones, I had no idea that the Google directory copied ODP.

[PhilC]

That's fair enough, macdesign, but DMOZ came into being for the purpose of sharing its data with any site that wants to use it, and that's still its purpose as far as I know.

*If* Google is dumping DMOZ clones just because they are DMOZ clones, then they would prevent the purpose of DMOZ from being fulfilled, simply because people want their sites in Google.

On the one hand, none of us want duplicate listings in the serps, and on the other hand, an engine shouldn't effectively prevent DMOZ from continuing to fulfill its purpose. It's a bit of a conundrum, isn't it.

[projectphp]

That's fair enough, macdesign, but DMOZ came into being for the purpose of sharing its data with any site that wants to use it
Really? I thought it was a response to how bad Yahoo was.

But who cares? I don't think that DMOZ reason for being, or why it started, or even its social contract has much impact on individual editors. As such, I don't think that any of the arguments you presented above makes much of a case for this issue, the clones targetted, doing anything to DMOZ that would cause its mortality.

I think you are throwing a lot of *ifs* on top of each other, Phil, and when one puts enough ifs together, you can get so far from where you started the conclusion is off.

If DMOZ editors really editted for the sake of the RDF dump and if Google was dumping clones, and if this caused clones to die anf if the only clone left was the google directoty and if editors could be convinced they were helping Google amke millions, , then your hypothesis may have been valid.

Personally, I don't see that as particularly likely.

[PhilC]

You appear to to be misunderstanding what I've said, projectphp. I didn't say that DMOZ would die, and I didn't say that DMOZ would die for the reason that I outlined. I said that that reason *could* cause DMOZ to die - and it certainly could.

"Ifs" are perfectly valid, and the one "if" that I used doesn't get away from where we started, as you suggested. The question was, "Is there anyone who can come up with one halfway decent reason why purging the index of duplicate content would have one iota of an effect on Dmoz?", and we didn't go away from the current supposition (not mine) that Google is dumping DMOZ clones. IF Google dumps the DMOZ clones, then the only sites left using the data will be Google, and possibly a handful of big sites. That's only one "if", and it reflect what some people think is the current situation, so it hasn't gone far away at all. The question then would be, what will the editors' response be? No "ifs" at all in that.

I may be mistaken, but I would think that most editors edit for the benefit of other people. They try to create as good a directory as they can for the benefit of other people. If (it's the same "if", not another one) other people are prevented from making use of their work, many or most of them may call it a day.

Incidentally, I don't know if DMOZ was created as a response to the quality of Yahoo! or not. I was around at the time, but I don't remember. However, it's purpose was to share its data, which is what I said ;)

[projectphp]

I said that that reason *could* cause DMOZ to die - and it certainly could.
Oh, ok. Apologies. I thought your hypothesis was something you believed likely, not just some wildly improbable possibility ;)

[PhilC]

Nope - it's not something I believe will happen, because, as I said, I don't believe that Google is specifically dumping the DMOZ clones. They may be, and I may be mistaken. If they are, or if they do, then it's something that would very likely make the editors stop and think about their positions, and they could decide not to continue working on the project for just a handful of big, profit-making sites.

[Marcia]

I don't believe that Google is specifically dumping the DMOZ clones. They may be, and I may be mistaken. I don't think there's any doubt there are common characteristics that could cause being filtered out.

If they are, or if they do, then it's something that would very likely make the editors stop and think about their positions, and they could decide not to continue working on the project for just a handful of big, profit-making sites.Phil, I doubt that the clones have very much to do with editors' motivations for volunteering.

Besides, if the clones are used as a RESOURCE by webmasters for their visitors, if it's sincerely that they'll continue to use them regardless of whether they're indexed or ranking.

Besides, the project would be a resource even if there were no clones at all, irregardless of the cloning capability being available.

[PhilC]

Common characteristics that have caught some DMOZ clones along with some other pages and sites, yes, but I don't yet accept that they are specifically targeting and dumping the clones - but I may be wrong.

You may be right about the editors, Marcia. You asked for any possible reasons why DMOZ could be affected, and I gave one. I didn't say that it *would* happen, but it could ;)