I'm working on a site that has been around for appr 3 years and generally ranks well for most desired terms. To get new pages indexed quickly the company put links to the new pages directly off the home page - which is effective and after only a few days the new pages appear in Google's index and the ranking at number 2 for the desired search phrase.

All is well.......until three days later when Google drops the pages from it's index - I've done a site: search for the url and it brings back no results - and a search for the actual url - and it brings back no results - it's quite literally no longer there. This keeps happening with all new pages added. There for a couple of days - ranking well - then dissapearing completely.

Also mysterious is that during this whole period Google webmaster tools does not indicate that the site has even been spidered - so further to this - how is the page being picked up anyway? So my questions are

1) How is the page being picked up if the site isn't being spidered
2) Why are the pages being dropped from the idex
3) How can I stop this happening

Would very much appreciate clarity on this as it's driving me crazy and I can't find any relevant info anywhere.

many thanks

Could the site have been banned?

Could the site have been down?

Are there any results for "site:url.com" and/or "site:www.url.com"?

Does any part of the meta say "noindex" or "nofollow"?

Is there a robots.txt and what does it say?

Have you used any 302 and/or 301 redirects?

Has the IP changed in the past few days?

Could the site have been banned? - no - good ranking across all category pages

Could the site have been down? - it could have been - but this is not a one off problem - it has happened consistently for every product page

Are there any results for "site:url.com" and/or "site:www.url.com"? - both of these produce the expected results - all indexed urls - with the exception of product pages

Does any part of the meta say "noindex" or "nofollow"? - no

Is there a robots.txt and what does it say? - definately not blocking the directory that the product pages are re-wrtten to (have double checked access in webmaster tools)

Have you used any 302 and/or 301 redirects? - not as far as I am aware - just the url rewrite

Has the IP changed in the past few days? - no

Hi thanks for the reply -

The site ranks well for category pages - but does tend to have a problem with product pages being indexed. They have an xml site map that is auto updated when a new product page is produced - I would say that 100% of category pages are indexed and about 10% of product pages. They rank for a few days and then dissapear even if they do appear on the site map - if they don't put a link off the homepage - they don't seem to get indexed at all.

There is one odd thing that springs to mind - they have a url rewrite for all product pages to make them SEO friendly - this is custom written rather than an off the shelf solution - do you think this might have something to do with it?

In terms of on the page stuff - if anything I would say they are in danger of being over optmised (although the category pages follow the same SEO practises as the product pages and these seem to do ok).

With regards to duplicate content - not an issue other than the url rewrite, but I'm pretty sure that it is only picking up the "re-written" url and not the original.

Can't think of anything else that might help!?

cheers

If you visit the site from the home page and drill down to a product page, is that URL in the Google Sitemap (XML) and the site's sitemap? Look and see!

When a user visits a site and selects a category and then a product the category is included in the URL. Is the category included in the URL of product pages?

When you say URL rewrite, what do you mean? (ie "mod rewrite" or other)

How many parameters are in product URLs?

Is a javascript used to create any URLs?

What happened to the old URLS? Are they now dead links or is there a 301 redirect to the new URL?

If you visit the site from the home page and drill down to a product page, is that URL in the Google Sitemap (XML) and the site's sitemap? Look and see! yes it is

When a user visits a site and selects a category and then a product the category is included in the URL. Is the category included in the URL of product pages? no it goes www.domain.co.uk/category.asp to www.domain.co.uk/directory/product.asp

When you say URL rewrite, what do you mean? (ie "mod rewrite" or other) I have just spoken to the client - it's not a re-write as such - the url's are re-written within the database using a VB script- they take the name of the product - hyphenate it and produce an asp page so the domain is www.domain.co.uk/directory/product-product.asp

He did say whilst answering this question that the page that produces the product html page relies heavily on server side includes


How many parameters are in product URLs? Just the one, a hyphenated product specific

Is a javascript used to create any URLs? - no

What happened to the old URLS? Are they now dead links or is there a 301 redirect to the new URL? as explained - there above are no dead urls or redirects - renaming of url's is done before html page is created

That's it? no solution? WAY TO GO BEU, all you did was have this guy running around wasting time and answering your questions and you didn't do a damn thing. Pathetic...

I registered just to say that.