Google Not Indexing the Directories Category

[krisval]

I just did my monthly check on a few DMOZ categories to see if any new sites were added. When I looked at the directories category:

http://www.dmoz.org/Computers/Internet/Searching/Directories/

I noticed it had no PR. I couldn't recall if they moved this to a new category and it simply wasn't reporting PR yet so I did a Google site search and it doesn't appear to be indexed.

site:dmoz.org Open Directory - Computers: Internet: Searching: Directories

or for a paticular listing that I know has been in this category for a long time.

site:dmoz.org Chiff

I checked the robots.txt file, headers for that page, and the previous page to see if there were any robot exclusions. I coudn't see any.

FYI. I saw this several days ago and decided to wait to see if it was crawled.
Is this Google targeting this particular category or was it simply moved and not indexed yet?

[I, Brian]

Hope this helps: Directories being Targeted by Google? (http://forums.digitalpoint.com/showthread.php?t=16937) at Threadwatch with discussion and original report at Digital Point with discussion: Directories category banned by Google? (http://forums.digitalpoint.com/showthread.php?t=16937)

[Marcia]

krisval, thanks for posting about this for us here. :)

I noticed not too long ago that in some other categories where I was looking for some things, the lower level categories had been re-organized and move around. I believe they do some periodic housecleaning around there. In that case, the categories could actually have PR, but just not be showing it on the toolbar at this point in time.

Have you noticed any changes in the category tree, or is it the same as it always has been?

[krisval]

Marcia,

I have visited that category in the past, but I honestly can not recall if it was in the same category structure. It is possible that it was recently moved. Maybe an editor can shed some light?

I did however notice this a few days ago and waited to see if it would get indexed before I posted. I would think with a PR7 page linking to it that it would at least get indexed by now. No Index, No Cahce, No PR.

Thanks for the info Brian that thread is dated 05-24-2005, 03:51 PM . This means that the category definitely should have been indexed by now. There is no robots exclusion on DMOZ that I can find for this category. No rel=nofollow, robots.txt, and header exclusions. I am beginning to think it has been targeted by G.

[Marcia]

There are other Directory categories that are just fine

http://www.dmoz.org/Shopping/Directories/

So for sure it isn't all directories, or all of those would also not be showing PR. Sounds like a reorganization possibility to me. Either that, or that one particular part of the directory may just be a problem area, which makes perfect sense.

[krisval]

Marcia,

I thought that coudl be the case, but as the other forum pointed out.

http://dir.yahoo.com/Computers_and_Internet/Internet/World_Wide_Web/Searching_the_Web/Search_Engines_and_Directories/Directories/

Is not indexed either. This certainly looks like it is possible that Google specifically targeted Internet Directories.

[Genie]

Marcia - there have been no recent changes to the structure of http://dmoz.org/Computers/Internet/Searching/Directories/ .

Also http://directory.google.com/Top/Computers/Internet/Searching/Directories/
currently has no PR either. Which does rather suggest a manual intervention by Google. Perhaps the idea is to discourage link-buying (in effect) from commercial directories?

[cbp]

I recall a discussion 9-12 months ago somewhere as the category at DMOZ for sites offering link exchange services was dropped from Google (its back now) ... I recall someone speculating that one (or more) link(s) in the category may have accidently broken the Google "don't link to bad neighbourhood's" guideline, so the page got dumped.

HOWEVER, it could also mean nothing. One of the categories I edit was dropped from Google for a few or more weeks a while back.

[Marcia]

krisval, the Yahoo Directories Category (http://dir.yahoo.com/Computers_and_Internet/Internet/World_Wide_Web/Searching_the_Web/Search_Engines_and_Directories/Directories/) is showing a healthy PR5 right now. Notice also that there are very few sites listed there.

But look at the Yahoo submission form for that category:

http://add.yahoo.com/fast/add?2014784

It's $299 per year recurring fee for commercial sites - and free submit is *mostly* intended for non-commercial, though some ecom sites have been known to be included that way. I seriously doubt that most of the directories around now will get in as non-commercial, and for a lot of the newer privately run directories out there now, $299 is quite a hefty fee when looking at cost/benefit possibilities - plus no guarantee and no refund if not accepted. Those have to rely on "other" avenues of marketing. ;)

Genie
no recent changesSo if it's not because of housecleaning and moviing things around, then it's a bug. Or maybe it's a new *feature*, depending on the perspective.

[Genie]

Now I remember the discussion you mention cbp. The suggestion about linking to bad neighbourhoods having triggered a penalty does seem more plausible than manual intervention.

[Robert_Charlton]

Could also be that they're just mixing things up a bit to raise the paranoia level. On the other hand, there's good reason to be paranoid about some of the directories out there....

[krisval]

Marcia,

That is odd. I could swear that just yesterday there was no PR and when I did a site: search in google, I couldn't find the page in the index. Today, I do a site: search and it is in the index. So that's good.

On the DMOZ side, never mind the PR, I was looking to see if the page was in the Index. As far as I can tell by using multiple keywords and the site: command, that page is not in the Google index. If you can find an example let me know. I would love for this not to be true. The IBL to this page is a PR 7 and it has had this structure since at least the end of May. It would be very odd that the page would not be indexed by this time. 3 Weeks.

By the way, based on my experience of setting up a new site, the PR dervide from submitting to about 10 or so directories is only worth about a PR 3 or on the outside a PR 4 to your site. So, if Google wanted to kill inflated PR, they should focus on link selling/site wide links not directories. Just my opinion. That is if they are targeting directories at all.

[Jmoney1172]

the Yahoo Directories Category (http://dir.yahoo.com/Computers_and_Internet/Internet/World_Wide_Web/Searching_the_Web/Search_Engines_and_Directories/Directories/) is showing a healthy PR5 right now.I still see gray bar but it seems to be indexed by G$ (http://www.google.com/search?sourceid=mozclient&ie=utf-8&oe=utf-8&q=site%3Adir.yahoo.com%2FComputers_and_Internet%2F Internet%2FWorld_Wide_Web%2FSearching_the_Web%2FSe arch_Engines_and_Directories%2FDirectories%2F).

As far as the Directories sub-cat in DMOZ goes... I recall seeing it dropped from the Google index back in early April - perhaps even March.
The suggestion about linking to bad neighbourhoods having triggered a penalty does seem more plausible than manual intervention.Indeed, it does. I can count the number of "useful" directories listed in this category on one hand. It's a shame because there are a lot of good resources listed in the ODP, Family Friendly, Non-Commercial, and Volunteer-Edited sub-categories - none of which are indexed by Google.

[krisval]

Bad Neghborhood Indeed, it does.

What do you base this opinion on? I don't see it. There isn't a high percentage of banned/possibly banned sites in this category.

Believe it or not....now this afternoon they Yahoo Cat is greyed out for me too. This morning a PR5. What the heck?

[Robert_Charlton]

Believe it or not....now this afternoon they Yahoo Cat is greyed out for me too. This morning a PR5. What the heck?

When my TV remote does that, I bang it on the table. ;)

[Marcia]

You know Google is gaslighting us with the toolbar. ;)

krisval, which toolbar version are you using?

[Jmoney1172]

What do you base this opinion on?Don't really have an opinion to base it on. Just agree with Genie and CBP that it's more plausible this category was penalized b/c of the listed "resources" instead of a manual intervention.

I mean this category isn't exactly booming with user-value. I would say there are a handful of directories listed here that can be considered comprehensive indexes of the WWW - the rest just seem outdated and no longer relevant to such a general topic. I think if an editor were to re-evaluate the listings in this category, he or she would remove or sub-categorize the not so relevant ones and Google would eventually re-index. OK - so I guess I do have an opinion/theory. :)

[krisval]

Marcia,

The latest one Version 3.0.123.2. English...Now that you mention it Marcia, I have seen the PR fluctuate with this toolbar ever since I installed it.

JMoney,

OK thanks. Just wanted to understand your thoughts.

Has anyone been able to find the DMOZ cat in the Index by using the Google site: command?

[Genie]

Not sure about that Jmoney1172. I don't think a page can incur a Google penalty simply for linking to a site that is out of date or a limited resource. As I understand it, the Google penalty for linking to "bad neighborhoods" is more specific. Google states:
Don't participate in link schemes designed to increase your site's ranking or PageRank. In particular, avoid links to web spammers or "bad neighborhoods" on the web as your own ranking may be affected adversely by those links.
http://www.google.com/webmasters/guidelines.html

In theory the Dmoz directories category could be penalised by linking to just one site seen by Google as a link farm or web spammer. We would expect that site to have no PR. http://directory.google.com/Top/Computers/Internet/Searching/Directories/ shows two sites at the bottom with no PR. One of them was recently removed from the ODP category. So it will be interesting to see what happens in the next PR update.