Cache Problems Growing for Directories?

[mixer28]

Less pages of the tier3 and tier2 directories seem to be cached by Google in the last 2 months. As I'm submitting sites its taking longer and longer to find directories with cached pages that are relevent. This can include established directories like Joe Ant etc

Out of 25 shortlisted directories I am finding I can only submit to 1/2 now as pages are just not being read anymore. Anyone else finding similar problem? Anyway we can get round it?

[sem4u]

I was going through a directory list the other day and found the same problem. The same goes for reciprocal link pages on sites. Some pages even have a PR value in the toolbar but no cache.

[mixer28]

yeah - this changes the classic seo model of link building for new sites to my mind.

For the last 3 years its been a great way to entrench new and existing sites but with even the more established directories struggling and reciprocal links lacking in effect I'm scratching my chin a little bit to figure out a solution.

One issue may be that directories need to make an effort to get links pointing to their actual listing pages - this could help as the pages then have atleast some form of external citation in Google's eyes?

[seoguyuk]

It is not just directories, many other non directory sites experienced this. In one of my sites, Google had shed almost 20'000 pages, which lead to drastic cut down in my traffic

[stoner3221]

I’m sure it’s not just directories but the effect of the Big Daddy update I have seen on directories starting around mid March has been in two directions. Many underdeveloped newer directories lost large numbers of cached pages. Larger well established directories now have more cached pages now then us operators ever considered possible with Google.

Personally I have a smaller niche directory that took a huge beating and two well developed long established general directories that have benefited immensely from the Big Daddy update with a large number of cached pages and the organic search results that come from having them.

My conclusion is Google wants a directory to be established, have a lot of content and the content must be of good quality. The new directory now is going to have to work a lot harder in developing it to be able to provide good link value for its submitter.

It used to take three years of hard work to develop a directory and I think we are back to that point again.

[Marcia]

It's the same phenomenon that's been hitting non-directory sites as well, with a limit to the levels in that will be crawled and indexed.

The truth of the matter, as far as directories are concerned, is that with a lot of the directories that have sprung up out there, there's a bogus link profile, with the majority of their IBLs coming forum posts and from linking to each other - and from sites that capitalized on the craze and/or were making profit from promoting them.

For many months people promoting directories (and makng profit from doing it, I might add, in a few different ways) were running around repetitiously doing multiple URL drops in forums all over, linking to the directory sites. At this point in time such links aren't worth squat, and haven't been for a while.

IMHO that was part and parcel, not of creating valuable directories for webmasters and surfers, but rather for getting the directories out and about for the purpose of buying and selling text links.

The people who were doing the URL-dropping may play wide-eyed innocent now, but to some extent we all have them to thank for pushing the envelope so far that Google had to take drastic measures to stop the epidemic. Thus - Big Daddy wielded the mighty rod and many who were and are innocent are feeling the sting who had no involvement and don't deserve it. No doubt, that's a big part of the reasoning behind the new "system" because it hits right in the bread basket and stops a lot of it from profiting algorithmically.

Oh, everyone may have been gung-ho at the time of that fad, except those who figured out what the overall scheme of it all was and saw the handwriting on the wall. No suprise what has happened - as they say, the proof of the pudding is in the eating.

Added:

Y'all might want to take a look back at this thread:

http://forums.searchenginewatch.com/...ead.php?t=6642

Note: I'm not implying that that is all of the problem and cause, because there's definitely more to it, but the non caching indicates dumping, and from the looks of it, it's redundancy and page stuffing that were targeted, though in some cases the baby is being thrown out with the bathwater.

[stoner3221]

I’ve just been sitting back watching and shaking my head for the past year and a half. Now that the ax has fallen and the links that put many website of marginal and poor quality in good positions are worthless; we can now listen to all the crying over it. Experienced reputable search engine marketers have been warning about this for over a year but have been ignored by far too many.

It’s unfortunate that many good websites with hard working devoted webmasters were also taken down with the Big Daddy ax. Search engines are just machines and far from perfect. Hopefully the problems that caused the good websites to loose cached pages will be corrected in the near future.

[obuwebco]

Yep i hear ya.

A directory site we built for a client about 8 months ago, that was steadily growing in usage. about 30k pages indexed, now shows 14 pages indexed in google.

Which is a real shame, no spamming links, no funny business. Just good old fashion sorting of businesses.

I am new to this forum, it is very well informed and knowledgeable one to boot. AND very pleased I found you. . .
But, I have just had the 'stuffing kicked out of me' as I(just a month old) started an article directory website without having all this information.

so, i am doomed before I even begun. Which is a real shame as I had hoped to make a living from it. I had big plans to be an internet marketer.

Joining forums and learning, reading from forums is all a waste now.
I will stick around to learn what works and what does not. as you mentioned about being in forums.

Right now, i am unable to start a new project. So, I can just give up before i waste my time trying.

You all been sincere in your comments and I am sure you most likely right.
I am no great webmaster(Im not a webmaster) by any means but willing to learn. But, I will have to rise up again another day

[stoner3221]

Originally Posted by goodluck222

I am new to this forum, it is very well informed and knowledgeable one to boot. AND very pleased I found you. . .
But, I have just had the 'stuffing kicked out of me' as I(just a month old) started an article directory website without having all this information.

so, i am doomed before I even begun. Which is a real shame as I had hoped to make a living from it. I had big plans to be an internet marketer.

Joining forums and learning, reading from forums is all a waste now.
I will stick around to learn what works and what does not. as you mentioned about being in forums.

Right now, i am unable to start a new project. So, I can just give up before i waste my time trying.

You all been sincere in your comments and I am sure you most likely right.
I am no great webmaster(Im not a webmaster) by any means but willing to learn. But, I will have to rise up again another day

Don’t give up on your project. I’m sure the near future will show some improvements in the area specifically that seems to be causing the problems. I’m not an expert but I have a blog post here that’s far too long winded to put on the forum that might help you understand some of what is going on. I strongly believe we will see some modifications and websites gaining back some of the lost cached pages.

[Cristian Mezei]

This problem is certainly not a directory targeted one.

It's all over the place and IMHO, directories as a specific niche of websites, have not been affected in particular.

[SEO1]

The problem deals more with links being devalued, as should have happened due to link spam. As stoner has posted, a few of us have stated that the link sprees webmasters have gone on in the past... are going to come back and bite them in the butt.....seems that has happened / is happening.

Where it was once thought to go deep... it now seems better to go wide... not deep....remember the old adage about keeping everything within one click?? now might make that an imperative.

Link to your sitemap page from the home page if you do not currently. Subdomains would seem to do well now more than ever considering the state of things.

Another consideration is microsites....something to chew on...

[PhilC]

It's nothing to do with the depth and one-click stuff. Many of the remaining pages of sites that have had most of their pages dropped are deep - I know of as many as 4 sub-directories and 3 clicks deep, while one-click pages were dropped. There is no pattern concerning the depth. If I'm not mistaken, that's what stoner3221 found.

And it's nothing to do with link sprees either. Sites that never did any link-building of any kind are suffering the same fate.

From what I can understand, it's to do with evaluating a site's IBLs and OBLs to decide how much of the site to index, and if a site doesn't score very well, some, many or most of it's pages are dumped to match the score. The evaluation concerns the types and quantities of links, and certain types are not being counted.

And Clint, there never was an adage to keep everything within one click, and it was never thought better to go deep

[stoner3221]

Originally Posted by PhilC

It's nothing to do with the depth and one-click stuff. Many of the remaining pages of sites that have had most of their pages dropped are deep - I know of as many as 4 sub-directories and 3 clicks deep, while one-click pages were dropped. There is no pattern concerning the depth. If I'm not mistaken, that's what stoner3221 found.

And it's nothing to do with link sprees either. Sites that never did any link-building of any kind are suffering the same fate.

From what I can understand, it's to do with evaluating a site's IBLs and OBLs to decide how much of the site to index, and if a site doesn't score very well, some, many or most of it's pages are dumped to match the score. The evaluation concerns the types and quantities of links, and certain types are not being counted.

And Clint, there never was an adage to keep everything within one click, and it was never thought better to go deep

PhilC you hit the nail right on the head. It will surprise a lot of folks to see what I see developing now in major directories. Pages 3 and 4 topics deep with good content having a higher page rank then the 2nd level and pages 6 levels deep rating as high or higher then second level. Topic depth means very little; topic content is the primary factor. If a topic is doing poorly then chances are it has excessive 404 sites or some poor quality submissions. Every topic page has to be dealt with like a single page website. Don’t expect it to rank highly but unless it is maintained well don’t expect it to rank at all.

[Cristian Mezei]

Some older directories don't have a problem, even if they have questionable (outbound) content. .

I think it's much more than just a statement of "IBL and OBL quality". It's about age, trust, and other things.

The stability of the IBLs might also have something to do with the crawling patterns (deeper or superficial).

Empty categories too.

But this is certainly not a directory-only problem. The indexed pages fever has(is) migrated(ing) to a lot of niches, even to very old quality websites.

[SEO1]

Phil

Had you read my post the reference to 'go wide' is advice. As for deep crawling I think you are mistaken there as well, it has been a mantra for a while for most marketers..

Next I doubt OBLs have much to do with things ...unless you play the link spam game...say links to a forum for geeks from a mortgage site. Mortgage site should get a penalty. If the forum was found to have particpated then they too should have the link filtered.


Next did you read your post. Your first two lines contradict themselves

It's nothing to do with the depth and one-click stuff. Many of the remaining pages of sites that have had most of their pages dropped are deep

It has nothing to do with depth, yet the next sentence says the pages dropped are deep..

Tis what I said...and the advice again

Go wide..Peace and goodnight.

[Marcia]

first two lines contradict themselves

No, they don't contradict if they're not pulled out of context and if they are read along with the rest of the paragraph, within the context.

First and second sentences:

It's nothing to do with the depth and one-click stuff. Many of the remaining pages of sites that have had most of their pages dropped are deep -

See, sentences have subjects and predicates and clauses, and if we read that with understanding of what the subject of the sentence is, then it makes perfect sense and there is no contradiction.

Read: Many of the sites that have had most of their pages dropped have many remaining pages that are deep. Interpret first sentence: in those cases, it has nothing to do with the depth and number of clicks.

Rest of sentences expressing the thought in context:

I know of as many as 4 sub-directories and 3 clicks deep, while one-click pages were dropped. There is no pattern concerning the depth. If I'm not mistaken, that's what stoner3221 found.

Very simply, what that is saying follows right along with the previous sentence - there are deep pages in sites that are 3 and 4 clicks deep that are NOT like the ones being compared with on the sites, one-click pages that were dropped - those deep pages weren't dropped.

It's the sites that have had many pages dropped - that is not saying that the deep pages were dropped in those cases - it's referring to the site, and then saying that deep pages weren't.

In spite of the slightly convoluted sentence structure, and one sentence that's fragmented, what's been said is perfectly clear. But now that we've got that straightened out, discussion can go on about a very good point that was made. Which I'm not 100% sure I agree with, but I'm all ears to hear more.

[PhilC]

Thank you Marcia

In a nutshell, Clint. Sites with deep and shallow pages have some pages remaining, and the rest dumped. Some of the remaining pages are deep and some are shallow. Some of the dumped pages are deep, and some are shallow. I.e there is no pattern concerning depth.

I know that this thread is about directories, but the problem doesn't just concern directories - it concerns all types of sites. Even so, I have a small niche directory that has been hit, and it's small enough to have only 12 pages left out of many thousands that were fully indexed. With only 12 pages left, it easy to see things. One of the things that I see is that a page's content doesn't make any difference.

The nature of the directory's listings is such that some of them are linked, most of them are not linked, some have descriptions (more relevant text on the pages), many of them don't. Of the 12 pages that are fully indexed, one is the homepage, one contains only links to lower cats but no listings (a one-click page), some have no linked listings and no descriptions, some have a few OBLs and some don't have any, and so on. In other words, it's just a general mix of pages without any discernable pattern of content, clicks, OBLs, whatever. So what I see with that site doesn't match what you said about yours, stoner3221. If anyone wants to have a look for patterns, it's www.forthegoodtimes.co.uk

The site was designed to be squeaky clean, and no link-building was done for it, other than giving it a few IBLs from my own sites to get it started. It was discussed in another thread, so there's no need to discuss it in detail in this one.

Putting what I see with that site together with what Matt Cutts wrote, I am led to the conclusions that (1) all the site needs are some good IBLs - the same will be true of many sites that have been hit, and (2) the remaining pages are not chosen according to some criteria of content or links, but the homepage and any other pages are chosen to make up the number that the site is allowed to have in the index. I even checked the order in which the site's pages were orginally crawled, and the order that they appear in the Sitemap, and there is no correlation between those orders and the pages that remain.

It's my opinion that Google now examines a site's IBLs and OBLs to determine how many of a the site's pages they will allow in the index. Certain types of links are not counted (e.g. off-topic IBLs/OBLs, reciprocals), and the score of the links that are counted is used to determine how many pages to allow. Your suggestion concerning "trust" etc. fits it very well, Expertu - it's about which links to count (trust) and which links not to count (trust), and the overall evaluation determines how many pages to allow in the index.

There may be other factors, such as site age, but I don't think that site age would be used, because it would mean disallowing up to date stuff. Imo, the new crawl/index function is about dealing with the link pollution that's been causing the deterioration of the index for many years.

That's my view of it, anyway.

[SEO1]

Marcia

Thank you foir clearing that up.

Has anyone looked at pages dropped vs. Google sitemap implementation?

Is there something in your Google Sitemap such as update frequency that might have something to do with pages dropped?

Or

Are you using a Google Sitemap?

As for inner pages having higher PR's I have seen this on my own site, and have a feeling it has a bit to do with intrapage linking and inbound links to the child pages.

In some instances if the highest PR site was a 7 pointing to a child page on my site, it would make that pages' PR higher, but not the overall site. This would seem to be bouyed if my home page highest PR IBL was a 6 or lower???

[PhilC]

Originally Posted by SEO1

Has anyone looked at pages dropped vs. Google sitemap implementation?

For the site I just mentioned, I looked at the Sitemap order and compared it with the remaining pages - no correlation. Also, the Sitemap doesn't specifiy an update frequency for any pages.