Google 300k vs Yahoo 2...?!

[Brown]

I know this kind of thing has been a problem for others because I have seen plenty of similar posts on numerous forums. Unfortunately, none of the info in these threads has helped so far. So any input is greatly appreciated…

I have recently been working on optimization for a large directory (300,000+ pages) that is mostly built with Perl and uses mod_rewrite to create static URLs. The majority of our entries exist as a profile page each with a unique subdomain. We also have a sitemap for these profile pages linked from the index. The site has been getting an increasing amount of traffic from Google (currently 1000+ unique daily) on a regular basis but is virtually invisible to MSN and Yahoo. For instance, Google is currently indexing about 294k pages but Yahoo is only indexing 2!

Before I started with on the project, there were two domains (.com and .net) for the site that both pointed to the same directory…a major problem, I know. However, that has since been changed so that the .com is redirected to the .net domain using a 301. I bring this up because Yahoo shows an index of about 1,700 pages for the old .com domain but only the 2 from the .net domain. This 301 was set up sometime last year so I would expect to see a more dramatic change in the number of pages indexed by now. So I’m wondering if Yahoo is just extremely slow to crawl these pages or if there may be some fallout from before the 301 was set up?

In the past week I have posted about 50 static, content rich pages with a sitemap linked from the index as an experiment to see how well these non-dynamic pages are indexed by Yahoo in the near future. I have also submitted several sitemaps to Yahoo for what that’s worth.

If we don’t start getting some traction with Yahoo soon, we are considering Yahoo’s paid inclusion program but are really not too keen on the idea of having to pay to get indexed when Google has already indexed hundreds of thousands of pages. Does anyone have any experience and/or opinions on paid inclusion that may be helpful?

Any other comments or advice on getting more traction from Yahoo (and MSN) would be greatly appreciated…Thanks!

~brown

[seoapprentice]

First off, I am not a Yahoo expert. I'm just a guy trying to figure out pretty much the same things you are.

My suspicion is that yahoo dislikes the use of sub domains. I suspect that is why we are banned in yahoo as we dropped shortly after switching from a directory/subdirectory structure to a subdomain.domain structure.

G and MSN are a different story as we are doing quite well in their indexes. For this reason alone we have not switched back from sub domains to subdirectories - fearing that we'll loose our G and MSN popularity. At our peek in the Y! index we only had a fraction of traffic from them, so it isn't worth jeopardizing our G ranking.

Perhaps an expert will chime in... it's been very very quite here lately. Has everyone given up on Y!?