Yahoo not indexing - any ideas ?

[Underpaid]

Just wondering if anyone else has experienced difficulty getting pages indexed in yahoo. I have 2 sites, both very similar in structure and style, but for some reason I cannot get more than 3 out of 25 pages indexed on my most recent one (3 mo old).

It does great in MSN and Google, no bad link neighborhoods, conservatively optimized, and a site map with links to all pages. Yahoo visits 1-3 times a week, and the 3 that are indexed rank very well.

The only difference I can see between this one and the fully indexed site is that the latter has a very meshed internal linking structure - almost all pages have text links to almost all other pages.

Any ideas ??

Thanks !

[Marc]

even i experienced the same problem with my site. around 800 pages indexed by Google, MSN all the pages but Yahoo has indexed only 1 page. :(

May be it is busy with its Tim update. Must wait and watch when this big guy will mercy upon our sites !!

Marc

[mcanerin]

Quick question to both of you: do you (or have you in the past) have more than one domain pointed at (specifically, parked) the site?

If so, does that domain show up for anything in Yahoo?

Ian

[Marc]

hi Ian,
my site was not pointed by some other url neither in the past nor now. All pure and safe.. :)

Marc

[Underpaid]

Ian,

No, no other domains have or are pointing to this one.

The puzzle continues.

[Marc]

My site is not parked. No other domains have or are pointing to this one. Even now only 1 page is indexed by Yahoo!. i'm worried..

Plz help..

Thanks
Marc

[Underpaid]

Marc, is your site an affiliate site ? It seems that this is mostly a problem with affiliate sites. And how long has it been online ?

[Marc]

Hi Underpaid,

mine is not an affiliate site. It was launched on March 15. Since then only 1 page indexed in yahoo!. plz explainn wats happening..

Thanks
Marc

[Carlos Chacón]

Quick question to both of you: do you (or have you in the past) have more than one domain pointed at (specifically, parked) the site?

If so, does that domain show up for anything in Yahoo?

Ian

Sorry, but I have the same situation…
What if that happens?

Is there’s a problem indexing on Yahoo if more than one domain is pointing at the same page? What about 301 re-directions?
:confused:

[Marc]

Hi Underpaid,

any improvement in your site pages being indexed by Yahoo!

My site's situation is same. Only the home page is indexed, even after 3 months of launching the site..

Help.. help..wats going on with my site :(

Thanks
Marc

[Underpaid]

Its too early to tell. 2 more pages just got indexed, but they may drop out again.

[Nacho]

I'm also experiencing the same thing with a site that has over 100K pages indexed in Google and 25K in MSN. Yahoo! only has 134 pages indexed. It has many backlinks (very popular site), so that’s not an issue. Slurp is very constant dropping by every day and then goes wild deep crawling. This behavior has repeated since the beginning. About 3 weeks ago, it hit over 40,000 pages on a single day.

I've talked to a number of people regarding this, because everything on the site is completely natural (or "white hat" as some call it). Every page has unique content. My perception and the census from what I've been hearing (and it includes other webmasters with millions of pages indexed on the other engines) is that there seems to be some type of error when Slurp drops off the pages in Yahoo!'s repositories, therefore never getting to the index. At first, I thought the pages might go through some deep analysis for its algorithms, therefore taking so long. Obviously, now I know that can't be.

Could this be on really big sites only??? :confused:

Perhaps...

According to the sites we manage, much smaller sites with 2k through 5k pages indexed don't seem to be going through these types of issues. The other thing is the smaller sites are older ones, while the example mentioned above is almost a year old. (I completely discard the term "sandbox" because I don't believe in it, but that's just me.)

Another possibility is, if there is no glitch as I said previously, then could it be possible that Yahoo! just has ONE BIG filter for newer sites that are really big and new? I could see the logic of that in terms of fighting spammers, but there are perfectly many legit (WH) reasons to put of a newer website with so many pages.

The only think I know that is for sure, is that I’m not alone and there are many of you out there experiencing the same on some of your sites (not all of them). Something must be happening. Is it a glitch or a perfectly explainable reason for Yahoo! doing this?

[Underpaid]

Nacho - With all of the Slurp action you are seeing, Im curious if your number of indexed pages remains fairly steady. My results from a site:domain query goes through periods of bouncing pages in and out of index. This may just be a normal shuffle, but I cant tell b/c this site is still new and small (35 pgs).

Your idea of a new site filter is interesting, but I havent seen any talk of that elsewhere. And Im not sure that would explain the steady growth of indexed pages, then a drop to 1 that Ive experienced. I would expect it to look more like the plateau seen for new sites in google.

Another thought that I have heard is that Y ! is simply not doing as much deep indexing. Perhaps they are not trying to grow their index as much as Google.

john

[Marcia]

How many levels deep does the directory structure go on the sites having problems?

[Underpaid]

My 'problem child' is a new site, so at this point, it only has one level.

[rustybrick]

This is a wide spread issue affecting hundreds of sites (i am sure more then 100s, because we are the only ones to notice).

[Marcia]

There were many sites a while back that reported disappearing from the index because they were hosted on "tainted" IP numbers. Do you happen to know if there are other sites in the same C-block or on the same IP (or even the same hosting) in the index and ranking?

[Nacho]

Nacho - With all of the Slurp action you are seeing, Im curious if your number of indexed pages remains fairly steady. My results from a site:domain query goes through periods of bouncing pages in and out of index. This may just be a normal shuffle, but I cant tell b/c this site is still new and small (35 pgs).Before they made the indexing update in April (http://forums.searchenginewatch.com/showthread.php?t=5010), we had a few thousand pages indexed and then it dropped. Now, the indexed size for that domain in question has bounced between 124 and 148 pages.
Another thought that I have heard is that Y ! is simply not doing as much deep indexing. Perhaps they are not trying to grow their index as much as Google.

john
Google has noted (http://googleblog.blogspot.com/2004/11/googles-index-nearly-doubles.html) and taken it a step further (http://googleblog.blogspot.com/2005/06/webmaster-friendly.html) to index everything on the web. In this quote, Shiva Shivakumar:It's a beta "ecosystem" that may help webmasters with two current challenges: keeping Google informed about all of your new web pages or updates, and increasing the coverage of your web pages in the Google index.I've bolded the words "all of your new web pages or updates" because this is what has made Google be what it was when it was born and what it is today... domination of world's information... ALL of it!

Think of it for a sec.

I've noticed that Yahoo! has really big duplication filters and many times a very similar page that only has one or two variances in the primary scoring on-page factors will be dropped. However, there is a perfectly logical explanation for it to achieve higher conversion by using the most relevant keyword(s) for every page. This could mean the difference between a 3.5% and a 5.0%, representing thousands of dollars. IMO, Yahoo!'s filters are a bit too strict. They should be allowing their index to grow more freely.
How many levels deep does the directory structure go on the sites having problems?Directory structures have their purpose, but it shouldn't be a limitation to indexing. Unless it makes the crawler go crazy, the in makes perfect sense. Our site that's having the indexing issues stays well inside the norm.

[Marcia]

I think we have to realize that we're talking about two different issues here:

1. Existing sites losing a number of pages from the index.
2. Inclusion of pages from brand new sites in the index.

As far as percentage of duplication is concerned, nowadays all the engines have to deal with a ton of swill that's flooding the internet in the form of scraper sites. Plus tons of duplication from affiliate sites using datafeeds - and auto-generated pages cranked out by software used for AdSense income, both on regular sites and blogs.

I don't think that's what the issue is with brand new sites of only 10-20-30 hand rolled pages. From what I'm seeing with new sites, Yahoo is being slow with including new sites in the index altogether when it first finds them.

[shor]

One of our sites has half a million pages indexed in Google. As recently as 2nd of June it has 300k+ indexed in Yahoo. Then over a period of around a week, we watched the index drop every day till we reached a low point of around 16,100 pages. After checking just then, we're back 'up' to 20k pages.

Our experience is very similar to Nacho - we've had Y!Slurp crawl this site for 10-15k pages a day during certain periods and we're all 'white-hat'.

As a counterpoint to this, another one of sites has doubled it's index in the same time period from ~300k to 600k+. Both are very 'mature' sites and have been around for years.

[Marc]

...

2. Inclusion of pages from brand new sites in the index.

...From what I'm seeing with new sites, Yahoo is being slow with including new sites in the index altogether when it first finds them.

I agree with u Marcia. From the things what i observed to my site, may b Yahoo! is slow to include new, big sites in its index.

hv 2 wait n c..

thanx 4 dat !!
Marc :)

[Marcia]

They aren't overly quick with indexing new small sites either. The currently 19 page site that's now under construction that I mentioned over here (http://forums.searchenginewatch.com/showpost.php?p=51402&postcount=12) was deep-crawled by Slurp on the 15th & 16th of this month, and so far only the homepage has been in the index even though a link to the site went up about 10 days before, which is how they found it.

Just to put an idea out about the big sites losing pages, isn't it possible that Yahoo is treating great numbers of near-duplicate pages similarly to what Google is doing, except that with Google they go into the Supplemental Index - while with Yahoo they may just be allowed to be filtered and slip out of the index?

Nothing to go by, just a thought and wondering if it isn't a possibility.

[JohnGalt]

I have had some recent action with one of my sites that was either penalized or dropped from Yahoo's index - only the home page was showing up in Yahoo's index for months. The site did undergo a redesign and then a few months later a url-rewrite (to address super long URL paramters/values).

I recently also had the client add a robots.txt file. They did not have one at all, and I could see that Slurp was requesting it over and over, as well as pages that were not in existence anymore. Don't ask, but it was decided not to implement 301s (against my recommendations). So, they added a very basic robots file (they did not exclude the old files and directories) but I figured that was better than no robots file at all.

Just before they added the robots.txt, they also added a sitemap that I had been pushing for.

All of a sudden on Friday I saw 3 pages in Yahoo's index instead of the one lonely home page which was all that was in there for months. One of those pages was the site map!

My best advice is to try different things - one at a time if possible, if you really want to try to pinpoint what may have helped. It is so hard to figure out what may be working if you change more than one thing at once.

Now I'm watching every day to see if more pages show up - the site has over 100 pages and tons of backlinks.

I'm starting to stray from the conspiracy theory thoughts I had been having, and I am starting to think Yahoo might just be a lot slower than the other SEs in updating new or redesigned sites.

There is another thread about the Yahoo indexing and/or penalty issue here (http://forums.searchenginewatch.com/showthread.php?t=5808).

[rustybrick]

A ray of hope, Yahoo! Launching New Search Index Tonight (http://forums.searchenginewatch.com/showthread.php?t=6378).

[critter]

A ray of hope, Yahoo! Launching New Search Index Tonight (http://forums.searchenginewatch.com/showthread.php?t=6378).

Does this mean more frequent updates or a change to their actual ALGO?

CRITTER

[rustybrick]

Does this mean more frequent updates or a change to their actual ALGO?

CRITTER
I know nothing more then what it reads there. We will have to wait and see.

[Marcia]

Has anyone gotten sites in or pages added to the index with this announced update of the index?

[ephricon]

Wanted to revisit this thread following Yahoo's new index launch. I've got significantly higher numbers of pages indexed. What about everyone else?

[Underpaid]

Has anyone gotten sites in or pages added to the index with this announced update of the index? A couple of pages were added, and a slight bump in rankings sometime between June 12-18. Since then pages continue to bounce in and out of the index, but overall rankings holding steady.

still underpaid :)

[Garyn]

Yahoo has aparently made a decision not to index certain types of sites that are not advertising clients of Yahoo or it's affiliated companies.

The result is limited search results for the public.

I believe this is a short sited policy that will drive searchers to other sites that offer unrestriced search processes which attempt to find the most relevant results.

Our sites rank very well on many of the most highly regarded search engines but we don't even show up on Yahoo. When Google was running their searches we were on page 1 for of most of our key search terms.

[Marcia]

Yahoo has aparently made a decision not to index certain types of sites that are not advertising clients of Yahoo or it's affiliated companies. And that assumption is based on ________ ?

How in the world could they possibly know which sites belong to their advertising clients, out of the tons of sites that launch daily?

[Garyn]

Prior to Yahoo terminating their search agreement with Google, site A was highly ranked ( page 1 for key site search terms ) on Yahoo.

After Yahoo search went inhouse the site does not even appear in their search universe although it remains highly ranked on Google & other major engines as well as on many independent search sites.

If site A provides the relevant information I need as a consumer, other search engines have no trouble finding and ranking it and Yahoo will not find it for me it seems reasonable to assume the site has been deliberately denied to me.

[Marcia]

How about a completely different algorithm?

remains highly ranked on GoogleApples and oranges.

[Garyn]

If their algorithm does not find the site in question they've got a problem.

After 5 years of operation this site has a 40+% visitor return rate, a 90%+ advertiser renewal rate, an average time on site of 5 minutes per visit, a 200% annual visitor growth rate since inception and over half the visitors typed in the URL or had it in their favorites.

I would say any algorithm that does not find the sites visitors like are indeed rotten apples.

[Underpaid]

I'm in a similar position, so I feel your frustration. But we also have to be realists. Not every great book gets published, not every great song gets on a DJ's playlist, and not every great idea gets a patent. They may all be worthy of a Top 10 position, but not getting there doesnt mean publishers, DJs and the Patent Office are all "rotten apples". There is limited room at the top, and many times the top also includes the mediocre.

I think the bottom line is that there are lots of people who are puzzled by Yahoo lately. There could be any number of reasons for the weird rankings. But IMHO, the best approach right now is to focus on doing the best you can for your visitors, and take what Google and MSN throw your way. Hopefully, our sites will recover soon :)

still underpaid

[Garyn]

I just ran a Yahoo search for the name of subject site. Several results came up for other sites where this site was mentioned in their text but the site URL did not appear at all. It appears not to be in their index even though over 500 other sites have inbound links to it according to Google, Alexa rankes it's market reach in the top 100,000 and the site receives 40,000 visitors and 2 million hits per month.

It apperars from this example that whatever Yahoo's algorithims are supposed to detect it's not sites relevant to what the searching public is trying to find.

[Marcia]

To get back in focus with the original topic in the first post of this thread:

http://forums.searchenginewatch.com/showpost.php?p=48953&postcount=1

Just wondering if anyone else has experienced difficulty getting pages indexed in yahoo. I have 2 sites, both very similar in structure and style, but for some reason I cannot get more than 3 out of 25 pages indexed on my most recent one (3 mo old). There's a difference between being indexed and ranking - and it looks like, from the context of the first post, that we're discussing whether or not sites are being indexed.

[Underpaid]

Marcia, that was my post. And yes, the issue is indexing. One site stays indexed, the other (4 months old) does not. When a few pages do appear in the index, even for just a couple weeks, targeted keywords on those pages rank very well.

[Marc]

Hello everyone,

For two of my sites, one of static HTML pages and the other a dynamic site with pages generating on-the-fly, Yahoo! indexed only the home page and not any other. Both are 4 months old.

My problem is: Why not other pages not been crawled by Slurp??

And for any of the keyword only the index page is shown.

How to come out of this problem?? Anyone facing the same problem??

Thanks
Marc

[edpudol]

I have 2 sites.. same 1 year old.. the first one got only 1 index page even do got high link pop other one I notice 2 days ago there are 1400 index pages but continuesly droping everyday. Right now there are 800 pages index only.

[seostruggler]

Marc - you have been hit by a manual review of your site - it has been deemed your site is not right for Yahoo by the editors and has therefore been stuck in the "only index the homepage of this site pot".

Cheers - bad news but you need to forget that domain on Y and get another.

[amye247]

I am jumpin' in...

My client has a huge site with over 350K pages in Google and MSN (each). There are over 10K in-bound links.

Yahoo however has only the home page indexed. It appears that this has been the case for at least six months.

What I do know-
1. the site is big
2. There are 2 URL's - 1 is without the www
2. the site is dynamically driven
3. IIS using a mod rewrite to handle URL's
4. I believe that this mod-rewrite uses a 301 to do this
5. no pages endings - all links point to xxx/rrrr/ffff/
6. it may be the forward slash - I ran a server check on some URLS
xx/rrr/ddd/ returns a 301 but
xx/rrr/ddd returns a 200

am I on to something here?

Amye

[edpudol]

Finally my 2 sites was index by yahoo not all pages but atleast good enough. I try to apply the blog and ping technique, hard to believe but my first site after 48 hours continuesly doing this, the site get 700 pages index and the second one I do it for 1 week and see yesterd 600 pages index and yahoo slurp still crawling.

I am not sure if this is coinsidence or not but the two site have only the main index, index before I do this for long time. So nothing lose guys if you try the said technique for atleast 1 to 2 weeks... my friend tried it also to her site with 6 months old that yahoo never index and now yahoo slurp crawling her's site pages very fast.

[Marc]

Marc - you have been hit by a manual review of your site - it has been deemed your site is not right for Yahoo by the editors and has therefore been stuck in the "only index the homepage of this site pot".

Cheers - bad news but you need to forget that domain on Y and get another.

If that is the case what should i do now, seostruggler. Is there any way out? Please help me..

Thanks
Marc

[dougedoug]

I have about 8 sites half affiliate and half for my company that do not even rank in yahoo. Currently I only have 4 or so sites that even have the entire site indexed. They rank everywhere except yahoo. What I see from yahoo now is alot of old information for indexed pages. I All of my sites have an ample amount of backlinks so they are showing up in results. Im going to try to get some backlinks to my secondary pages and my site map to see what that does. If I see any results, i will post them there. It just stinks how yahoo has about 30% of the search market, but they are getting so judgemental with who shows up there.