Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engines & Directories > Google > Google Web Search
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 03-23-2006   #1
docprego
 
Posts: n/a
Why have 99% of my indexed Google pages dissapeared?

It took Google quite a while to show up and start indexing my site. After about a month I had about 340 pages indexed, which for my site was terrific representing almost all of the important pages.

Suddenly and I am not sure exactly when, within the last couple of weeks my 340 pages indexed has dropped to only 1, the home page.

This search; site:mysite.com

Gives the following results; "Results 1 - 1 of about 4 from mysite.com"

The 4 results all point to the same page (my home page) but with slight variations, here they are:

"mysite.com"
"www.mysite.com"
"https://www.mysite.com"
"www.mysite.com/index.php"

I do not fully understand the implications of the Big Daddy update, is this a temporary problem being caused by this update? How can someone possbily find out what is wrong?

Any help or ideas would be most appreciated!
  Reply With Quote
Old 03-23-2006   #2
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
Quote:
"mysite.com"
"www.mysite.com"
"www.mysite.com/index.php"
Problem right there - never confuse a bot. Pick the one you want to use and use 301 redirection to redirect the others to that one. And ONLY link to that in your internal linking and with links from other sites that you request or submit for.

Quote:
"https://www.mysite.com"
https should not be indexed at all. And why is the root index page "secure?"

.htaccess can be your best friend sometimes. Get that straightened out and you'll probably see the situation ironed out - unless there are other problems, but that looks like it could be it.
Marcia is offline   Reply With Quote
Old 03-23-2006   #3
docprego
 
Posts: n/a
Thank you. I have a 301 redirect which takes all non-www pages to www pages. But I do not know how to get rid of those other pages. All I would like to have is "www.mysite.com."

What code would I add to the .htaccess file to accomplish this?
  Reply With Quote
Old 03-23-2006   #4
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
docprego, I know enough about .htaccess and mod_rewrite to copy and paste.

I'll see if I can ping a member who will probably know.
Marcia is offline   Reply With Quote
Old 03-23-2006   #5
seomike
Md_Rewrite Guru
 
Join Date: Jun 2004
Location: Dallas, Texas but forever a Floridian!
Posts: 627
seomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to behold
Hey docprego, This code should fix up your canonical issue
Code:
RewriteEngine On

#make non-www & non-ssl redirect to www version
RewriteCond %{SERVER_PORT} ^80
RewriteCond %{HTTP_HOST} ^yourdomainhere\.com
RewriteRule ^(.*)$ http://www.yourdomainhere.com/$1 [R=301,L]

#make non-www & ssl redirect to www version
RewriteCond %{SERVER_PORT} ^443
RewriteCond %{HTTP_HOST} ^yourdomainhere\.com
RewriteRule ^(.*)$ https://www.yourdomainhere.com/$1 [R=301,L]

Thx for the heads up Marcia, Glad I could be of help!
seomike is offline   Reply With Quote
Old 03-23-2006   #6
docprego
 
Posts: n/a
I greatly appreaciate your help but am confused.

I want only my "www.mysite.com" to be indexed. I already have a 301 in place to make all non www pages load up the www version. How do I eliminate these links:

"https://www.mysite.com"]
"www.mysite.com/index.php"
"mysite.com"

Thank you.

BTW, here is my htaccess code for the 301 just in case:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^mysite\.com [NC]
RewriteRule ^(.*)$ http://www.mysite.com/$1 [L,R=301]

Last edited by docprego : 03-23-2006 at 07:25 PM.
  Reply With Quote
Old 03-24-2006   #7
docprego
 
Posts: n/a
Another puzzling part of this; Google shows up daily and spiders as many as 50 of my pages. Yet still nothing shows up as indexed. It is this bot that visits: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Can anyone help me figure out what the problem is here, I am going insane over this! Can I contact someone at Google?

Thanks.
  Reply With Quote
Old 03-24-2006   #8
vayapues
10 kinds of people in the world. Those who know binary numbers, and those who don't
 
Join Date: Jan 2006
Location: Salt Lake City
Posts: 322
vayapues is just really nicevayapues is just really nicevayapues is just really nicevayapues is just really nice
hah, great code snippit. Thanks for sharing.
vayapues is offline   Reply With Quote
Old 03-24-2006   #9
seomike
Md_Rewrite Guru
 
Join Date: Jun 2004
Location: Dallas, Texas but forever a Floridian!
Posts: 627
seomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to behold
Quote:
I want only my "www.mysite.com" to be indexed. I already have a 301 in place to make all non www pages load up the www version. How do I eliminate these links:

"https://www.mysite.com"]
"www.mysite.com/index.php"
"mysite.com"
Well if people are linking to you using those urls then the 301's are going to have to tell the spiders that the www.mysite.com is the core domain. Once Google knows the correct core url, they'll show it in their SERPS.

you'll have to ammend the code above to fix the index.php problem.

####### fix for www.mysite.com/index.php ###########

Add this rule on top of the other 301 redirect rules

Code:
#Make this the first rule
RewriteRule ^index\.php$ http://www.yourdomainhere.com/ [R=301,L]

###### fix for mysite.com ####################
you have that already in place



###### fix for https://www.mysite.com ####################

As for the ssl problem. Do you have pages behind ssl? because the engines shouldn't be indexing that. If they show up as an ssl link and there is no certificate installed then the ssl layer warning will pop up in the users browser if this is the case just ammend the code I gave you so it looks like this.

Code:
#make non-www & ssl redirect to www version
RewriteCond %{SERVER_PORT} ^443
RewriteCond %{HTTP_HOST} ^(www.)?yourdomainhere\.com
RewriteRule ^(.*)$ http://www.yourdomainhere.com/$1 [R=301,L]
seomike is offline   Reply With Quote
Old 03-24-2006   #10
seomike
Md_Rewrite Guru
 
Join Date: Jun 2004
Location: Dallas, Texas but forever a Floridian!
Posts: 627
seomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to behold
Quote:
Another puzzling part of this; Google shows up daily and spiders as many as 50 of my pages. Yet still nothing shows up as indexed. It is this bot that visits: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Can anyone help me figure out what the problem is here, I am going insane over this! Can I contact someone at Google?

Thanks.
There is definitely a lag from crawl to cache. How long of a lag I really don't know, but as Google is updating their data centers you may see different caches depending on which data center you are visiting
seomike is offline   Reply With Quote
Old 03-25-2006   #11
docprego
 
Posts: n/a
Thank you but when I try some of those pieces of code it is not working. I get an error message saying something like that redirect can never work.

I want to load up www.mysite.com instead of www.mysite.com/index.php

Also I am unclear on how to get Google to drop the secure https page from their search results. The home page on the site:www.mysite.com search is coming up as an "https://www.mysite.com" result. The site:mysite.com search also results in only the homepage coming up but it is "http://mysite.com"

So how do I get Google to return a result for my homepage of http://www.mysite.com?

Thanks this is all so frustrating and confusing!

P.S. I still have no indexed pages, just the home page which has its own problems I mentioned above.
  Reply With Quote
Old 03-29-2006   #12
docprego
 
Posts: n/a
I am still facing the same problem, no pages other than the home page are indexed.

Can anyone help me?
  Reply With Quote
Old 03-29-2006   #13
sootledir
directory.sootle.com
 
Join Date: Mar 2005
Posts: 146
sootledir has a spectacular aura aboutsootledir has a spectacular aura about
How old is the site? If it's relatively new and has a low PR, it could be a long time until all the pages are indexed.
sootledir is offline   Reply With Quote
Old 03-29-2006   #14
docprego
 
Posts: n/a
Hi,

The site went online in December 2005. Google took about a month to show up and then I had over 340 pages indexed. Suddenly about a month ago it dropped to my homepage only. Please look at my URL in my profile and take a look at the results Google returns for the site:www.mysite.com or mysite.com searches.

The only modification I have made is that AFTER all my indexed pages dissapeared I set up a 301 for non-www pages. After this Google did drop the mysite.com version of my site from their index, but https://www.mysite.com and www.mysite.com/index.php remain in addition to the normal www.mysite.com indexed page. Could this be the cause of my problems? And if it is, how do I get Google to drop the https page and the www.mysite.com/index.php page?


Thank you.

Last edited by docprego : 03-29-2006 at 11:27 AM.
  Reply With Quote
Old 04-18-2006   #15
docprego
 
Posts: n/a
Update:

Google added back in 46 of my pages about a week ago, then those too mysteriously disappeared. I am now back to only my homepage being indexed plus the https version of my homepage. I tend to think that having the https and http versions of my page being the only 2 pages that show up in google's results probably has something to do with my problem.

Any ideas? Please look at my site and tell me what I can do to resolve this.

Thanks.
  Reply With Quote
Old 04-18-2006   #16
glengara
Member
 
Join Date: Nov 2004
Location: Done Leery
Posts: 1,118
glengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud of
*Any ideas?*

Look at the text-only cache and you'll see most of the text is from your manufacturers list, your pages may be being seen as mostly duplicate content.
glengara is offline   Reply With Quote
Old 04-18-2006   #17
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
Right at the top of the homepage is a link to sign in. That link is going to an https:// page - it shouldn't. That sign in page should be on a regular http:// page and only after they've sign in should people go to secure pages. Bots can't sign in, but the link to a secure page isn't helping anything, being the first thing hit.
Marcia is offline   Reply With Quote
Old 04-18-2006   #18
docprego
 
Posts: n/a
My site runs on OS Commerce. All sites using this software will have that link on the homepage unless it is specifically removed. It is supposed to be a secure page. Many sites that use OS Commerce are indexed just fine by Google.

At this point I am at a complete loss as to how to solve this problem.
  Reply With Quote
Old 04-18-2006   #19
clint7
Member
 
Join Date: Apr 2006
Posts: 11
clint7 is on a distinguished road
This may not be your problem. If you take a look at some of the other forums, there's a large and growing number of people reporting exactly the same issue: vast numbers of lost pages.

It seems that this may all be part of the painful Big Daddy transition, which, either by flippant design or serious mistake, has caused a whole host of problems to lot of legitimate websites. Most of the problems seem to stem from the fact that Big Daddy was kick-started from a stale index -one that probably pre-dates your site, for example.
clint7 is offline   Reply With Quote
Old 04-18-2006   #20
docprego
 
Posts: n/a
Well one thing I did do was remove the manufacturers box from my home page, that removed a whole lot of identical text which would have shown up to the search engine spider on every single one of my pages. I think that might have been a contributor if not the cause of this problem. Thanks glengara for that eye opener.

Interestingly I noticed that when my 46 pages briefly came back most of them were links to my categories pages which was fine BUT the description of every single one was the same. The description was the exact text that glengara pointed out to me as a potential problem. I just didn't put 2+2 together, I am hopeful that this could be the culprit. I know removing all that identical text certainly can't hurt.

Any further ideas would be great. Thanks!
  Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off