You've heard it all before: company URL vanishes from index

[Tooooons]

RE http://www.moviemusic.com/
From last week to this week, this is from our referrer log:

LAST WEEK
50% http://www.google.com/
13% http://search.yahoo.com/
9% http://www.google.ca/
5% http://search.msn.com/
4% http://www.myspace.com/
3% http://www.google.co.uk/
3% http://www.google.com.au/
3% http://www.emulek.com/
3% http://www.google.de/
3% http://216.239.39.104/
2% http://aolsearch.aol.com/

THIS WEEK through wednesday
55% http://www.google.com/
9% http://www.google.ca/
6% http://p085.ezboard.com/
5% http://club.xialala.com/
4% http://www.google.com.au/
4% http://www.google.co.uk/
3% http://www.myspace.com/
3% http://www.google.de/
2% http://www.emulek.com/
2% http://216.239.37.104/
2% http://www.neopets.com/
1% http://aolsearch.aol.com/


Notice Yahoo! and MSN are completely missing in this week's log. Searching for "moviemusic.com" yields zero results these days at Yahoo, MSN, Overture, AlltheWeb.

What causes something like this to happen? It looks like a "banning" to me, but we are the last company to try and fool search engines. It is my personal belief that "if you build it, they will come", so we've concentrated on building a unique website with a strong focus. For the last 5 years, our traffic and sales grow with very little marketing effort.

In 1999, I remember a test we ran with my DNS guy that involves the following mirror: http://movie-soundtracks.moviemusic.com/

We left this address on in the DNS server, but never publically marketed it. Might this be the problem? I would have thought if this were a problem, it would have been a problem years ago. Google essentially ignores it.

Other than that, no "optimization" enhancements have been made in the last few years.

So, any ideas among you on how to get the URL back into the Overture/Yahoo search engine? I've messed with MSN's and Snap's new search sites, and things are fine there.

Thanks for being nosey here. Appreciate your feedback,

Tooooons

[seomike]

22,500 pages indexed in google.com

0 in Inktomi recepticles.

Man you must have really pissed them off.

This is probably a long shot but...
Do you have anything that blocks bots used to strip email address or dowload pages in mass from downloading your site? I'd make sure that you aren't blocking any good bots.

[Joseph Morin]

This happened to me for a major ecommerce site, only it was booted out of Google, Yahoo and MSN all within a matter of weeks of each other, couldn't figure it out and traffic dropped 95%.

Used a user switching agent in Mozilla Firefox to emulate the various search engines and found that the server's software was redirecting clients, connecting with specific user agents of various robots, to a relative tracking URL. This is causing an infinite amount of redirects, forcing the spider to "give-up" and drop the offending pages from its index. Notably the home page.

Turns out that a programmer had accidentally loaded the wrong code in the homepage of the site during a refresh causing the incident.

[Joseph Morin]

Oh, and welcome to Search Engine Watch Forums Tooooons...sorry to hear about the problem.

[Marcia]

The first thing is to go to Yahoo's guidelines, and you'll find that mirror sites aren't acceptable. Something may apply even if it isn't deliberate and that's where to start, to remove any violations that may be found.

What's on the subdomain could be considered a mirror site, accidental or not it needs to go. But here's something that relates to what a lot of people are seeing nowadays, and having trouble over - Yahoo's handling of redirection.

Yahoo search for an exact phrase, copied and pasted from the first paragraph, actually from the mirror_copy subdomain:

Yahoo search for exact phrase (http://search.yahoo.com/search?fr=slv1-&p=%22order+today+from+our+extensive+catalog%22)

OUCH!!

Now, checking the headers for the URL that's appearing in the Yahoo search:

Server Response: http://film.coolbegin.com/g.asp?id=6658 Status: HTTP/1.1 302 Object moved Date: Fri, 08 Oct 2004 03:02:42 GMT Server: Microsoft-IIS/6.0 Location: http://www.moviemusic.com/ Content-Length: 147 Content-Type: text/html Set-Cookie: link6658=6658; domain=.coolbegin.com; path=/ Set-Cookie: ASPSESSIONIDASSQRCTA=CPMGHALCHNNPCELLDNJNONEA; path=/ Cache-control: private

That's from the Search Engine World Server Header Checker (http://www.searchengineworld.com/cgi-bin/servercheck.cgi)

Then check it here:

Rex Swain's HTTP Viewer (http://www.rexswain.com/cgi-bin/httpview.cgi)

So in addition to what could be construed as a mirror site on the subdomain that needs to go, you're dealing with a 302 Redirect (and Session IDs) - which is what I'd get rid of ASAP. I've seen a site PAY for advertising and have their page replaced for the search at Yahoo by the advertising page linking with a 302 redirection for the little button banner.

If the site got by for a long time with the "accident" in place, it's anyone's best guess which has now caused the site to slip out. While either can be a candidate for the cause, my vote would be with the 302 simply because it's such a prevalent problem at this time, even to the point of there being cases of deliberate sabotage that's affected Google listings as well.

[Tooooons]

Thanks Marcia. I figured the subdomain mirror may be an issue, but kept stumping myself because "www.moviemusic.com" and "moviemusic.com" are in the same situation. Perhaps search engines let a "www" DNS setting pass and not any others.

I do see the issue with the 302 redirects and session IDs, but I am uncertain as to how this might be a problem that I am able to fix. These other sites do not belong to me. Is there something on my server I can do?

I'll do some more research and see what I can find. Thanks for your attention. This is baffling, yet also a very big issue in this technology.

[Marcia]

Toooons, I think overall we have to look at the root of why it's being done, who's doing it and what type of sites they are to even begin to figure it out.

Looking at that page's source code in In Internet Explorer:

view-source:http://film.coolbegin.com

(Added: copy and paste into the addy bar minus the hyperlink.)

I'm at a loss to know what people can do. I was contacted by a client this week about scraped site pages showing up. Apparently some site in an Eastern European country is scraping pages left and right from all over, they're in the SERPs at Google and there is absolutely no contact information for those sites. And that isn't an isolated case at all, it's happening a lot.

The only thing I do know is that a lot of what I've personally seen with sites is that they're involved in affiliate programs, and the abundance of the worst I've seen has been PPC affiliate pages. I have no idea if or how much that relates to the amount of fraudulent PPC clicks we're reading about.

I think all we can know is that it isn't being done without some reason behind it that's making $$ for people anyway they can get it. Not only by hijacking pages in SERPs, but with drive-by installations of spyware and BHOs on computers as well.

[Tooooons]

Nothing is an easy fix, that's for sure.

I want to know more about this, and here are my first observations.

The server header checker for the site that is redirecting via 302:
(http://film.coolbegin.com/g.asp?id=6658)

================================================== =
Server Response: http://film.coolbegin.com/g.asp?id=6658
Status: HTTP/1.1 302 Object moved
Date: Fri, 08 Oct 2004 04:48:28 GMT
Server: Microsoft-IIS/6.0
Location: http://www.moviemusic.com/
Content-Length: 147
Content-Type: text/html
Set-Cookie: link6658=6658; domain=.coolbegin.com; path=/
Set-Cookie: ASPSESSIONIDASSQRCTA=IGBHHALCGEDCGONINLHCGNLJ; path=/
Cache-control: private
================================================== =

First, why the "Location" property says moviemusic.com is way beyond me (all the redirect stuff is happening in the g.asp file on coolbegin.com) and is probably causing the problem with the webcrawlers.

If only the webcrawlers would take into account the URL in the "Server Response" property and leave out the "Location" property, a lot of headaches would go away methinks. But I am not an expert, so I could be off base with this simplistic suggestion.

There's nothing I can do except to get in touch with the site and ask one of two things: 1) remove the redirection link to moviemusic.com, or 2) ask the redirection script to use permanent redirect 301 instead of temporary redirect 302. Right?

Considering what Marcia said about phantom websites and a possible inability to contact them for assistance, it appears to me that the search technology companies need to accept the burden of this problem and figure out a new way around it instead of completely removing search-compliant websites (doing so actually makes the web less attractive as good content is essentially wiped out, unable to be found).

Any more thoughts? Meantime, I have sent an email to the coolbegin.com webmaster.

[Tooooons]

The freakin' irony of this whole thing is that "coolbegin.com" with its 100s and 100s of subdomain keywords, has +/-100,000 links in Yahoo's web index.

I have 1 or 2 subdomains, including "www".

All it takes is one of their links to moviemusic.com, bad redirect and all, and my website is banned?

This is a flaw. Out for the night.

[Marcia]

Tooooons, we can't know 100% for certain about your site, in the final analysis only Yahoo would know that for sure. But these are the things that have to be looked at, especially with all that's been going on lately.

Here's your site once again Toooons, found at Yahoo

http://www.fastcomeurope.co.uk/022001/film/moviemusic.htm

Notice the difference if you click on links "normally" or with right clicking and opening in a new window. There's more, this is just a start with what can be found when digging around about missing sites. And a *lot* of sites are turning up missing now with people not understanding why.

In your place I'd get in touch with Yahoo about your site that slipped out and see if they'll be able to help you get restored. The subdomain left up in error is one thing, but I don't see how you could take the blame for other things. Unless there's some other reason - which there may be, but without asking them there's no way to know.

Added:
Another thing to check out - with templated sites, it can't hurt to check and see how unique pages are from each other and what percentage is duplication across many, many of the site's pages. There are people who suspect that there are Google problems over this so it is worth looking into regardless - for any search engine.

[Tooooons]

Marcia, I checked the server headers for the fastcomeurope.co.uk URL and they check out OK:

================================================== =======
Server Response: http://www.fastcomeurope.co.uk/022001/film/moviemusic.htm
Status: HTTP/1.1 200 OK
Date: Fri, 08 Oct 2004 15:40:27 GMT
Server: Apache/1.3.31 (Unix) FrontPage/5.0.2.2623 AuthMySQL/2.20
Last-Modified: Thu, 24 Jan 2002 10:23:49 GMT
ETag: "c82983-27a-3c4fe0b5"
Accept-Ranges: bytes
Content-Length: 634
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
================================================== =======


No redirect 302s, no session IDs, etc. The odd thing about this result is there is no "Location" property value. That must be a property assigned in the redirect script.


I have tried contacting Yahoo, but it is my experience (and most everyone else's), that Yahoo very very rarely replies due to an overwhelming number of customer service requests. Of course, I'd love to know exactly why my site has been removed according to Yahoo. Perhaps in the future, these search sites will be legally accountable for explaining why a company has been dropped from the index. Right now, no such thing exists, so I have to hope against hope that Yahoo will reply to my email.

This is the page I've used to try and contact them:

http://add.yahoo.com/fast/help/us/ysearch/cgi_feedback

Good luck to me.

Anyone have good advice on how to contact a Yahoo customer service person who can actually help? Thanks.

[strategicrankings]

Hi Tooons.

The problem you are facing has been discussed in length lately on another popular SEO forum. You are actually experiencing what is called the "duplicate content penalty". Actually it is not really a penalty, but a basic optimization inherent in all well structured database. Would you allow someone to insert duplicate entries for an employee in your employee database? Of course No, just to prevent redundancy.

What is happeing to you is the same thing. Yahoo & MSN are seeing your site and film.coolbegin dot com as duplicate and is dropping your site.

The webmaster at the offending site may be or may be not doing it intentionally. I would advice that you contact them to ask them to remove the culprit codes.
Don't want to be too pessismistic here, but the same may happen at any moment for your Google results. and your site will be found nowhere on the major engines.

Contacting Yahoo! will not solve your problem, since its may be clear for them that there is duplicate content.

So sorry for this.

Riley

[strategicrankings]

When this was discussed on the other forum, i remember a guy got a reply from Google saying that this situation was within their guidelines so they would not deal with it.

Noone is really protected from this.

Riley

[Tooooons]

This is a major flaw and has no logical value.

Essentially, the web experience will get worse as a result of this penalty.... meanwhile, search engine companies claim they are working hard to make the web experience better?

Death to the internet. I hear the toll bells already.

[Marcia]

Of course there's a duplicate content problem, but the problem is that too often now this is being done deliberately, and with off-shore companies it's all too common that there's no way to make contact with the offending site. Therefore, there is no remedy unless the search engines act.

Google and Yahoo don't handle things the same way, AFAIK. Take a look at item 4 here:

http://add.yahoo.com/fast/help/us/ysearch/cgi_reportsearchspam

What does that tell you?

[Tooooons]

Not believing at all what strategicrankings has suggested, I decided to run some tests. Results: my site will be fine in Google's index, as google doesn't even capture this instance of "content duplication" like Yahoo has.

A few postings back, Marcia searched for an exact phrase from my site and came up with these redirected links that are causing problems, most notably the link from film.coolbegin.com.

I have tried to replicate what Marcia did with Yahoo on Google.com, yet the link from film.coolbegin.com cannot be found at all in Google's results. In fact, none of the results found in Marcia's Yahoo search were found in Google's search.

I was hoping to find the film.coolbegin.com link on Google so I could report the site for "content duplication," but Google is on top of things. Here is their spam reporting URL for anyone interested:

http://www.google.com/contact/spamreport.html


Google is ahead of the game. Yahoo will sink fast if it cannot overcome these hurdles.

[Tooooons]

Perfect, Marcia! That's exactly what I was going to do next... search for Yahoo's spam-reporting page so I can report the film.coolbegin.com URL that is duplicating content. I'll do that now.

I still think the burden rests on the search engines to get this right. Google looks like it's getting somewhere because I can't find the link the same way I can in Yahoo. Yahoo's technology must figure out how to correctly read the server headers and process redirects before banning innocent domains.

[strategicrankings]

Don't want to be too pessismistic here, but the same may happen at any moment for your Google results. and your site will be found nowhere on the major engines

Hi Tooons, see that i mentioned MAY happen, i never suggested that i will actually happen.

Great that you did the check. Nice to note that the offending site has been dropped (instead of yours) by Google.

But there are cases where the original sites have been dropped by google, instead of the offending ones.

[Tooooons]

I don't think I meant to infer that Google dropped the offending site. coolbegin.com is all over the Google index. What I believe my test suggests is that Google knows what is true content duplication and a simple case of a "bad" site redirect. Yahoo is obviously having problems distinguishing between the two, whether it can't read the server headers right or it can't process redirects, whatever it is, and ends up penalizing innocent websites. Yahoo needs a serious modifcation, as this is their fault.

[Marcia]

No, Google doesn't necessarily know and there have been a lot of problems with Google with 302s - and hijacked pages. Pages may not be in Google because of lack of links. Yahoo doesn't handle 301s the "expected" way as well. But they are aware of it. And the two of them do deal with sites differently.

[Tooooons]

In the broader view, Google may be having problems, but I don't know them. I know this case only, and it appears Google knows the difference here. Yahoo does not.

I decided to do some more checking to see where any changes are coming from that might have sparked this issue last week.

So far:

1) The moviemusic.com website has not changed in any major way that would affect its search engine compliance. Even the mirror "movie-soundtracks.moviemusic.com" has been around for 5 years.

2) Using web.archive.org, it appears film.coolbegin.com has had the g.asp redirect script in place since at least June 2002. I would consider this an indication that a major change has not happened with them.

So, if we are even discussing the right issue, the only change that could have possibly been made to facilitate the removal of our domain from Yahoo web search, is within Yahoo's company. No one violated anything. Yahoo implemented flawed technology here.

I believe I have been barking up the wrong tree by contacting the film.coolbegin.com website. They've done nothing wrong at this point.

Doubtful, but it would be nice to get a reply from Yahoo regarding this.

[ThouShaltSeo]

I think you will see a change this or the next update with Yahoo. http://www.ysearchblog.com/files/wmw2004/search-friendly-design.ppt
some people who emailed Google reported that they (Google) see no problem and there's nothing to fix. We must all be delusional

In the broader view, Google may be having problems, but I don't know them. I know this case only, and it appears Google knows the difference here. Yahoo does not.

I decided to do some more checking to see where any changes are coming from that might have sparked this issue last week.

So far:

1) The moviemusic.com website has not changed in any major way that would affect its search engine compliance. Even the mirror "movie-soundtracks.moviemusic.com" has been around for 5 years.

2) Using web.archive.org, it appears film.coolbegin.com has had the g.asp redirect script in place since at least June 2002. I would consider this an indication that a major change has not happened with them.

So, if we are even discussing the right issue, the only change that could have possibly been made to facilitate the removal of our domain from Yahoo web search, is within Yahoo's company. No one violated anything. Yahoo implemented flawed technology here.

I believe I have been barking up the wrong tree by contacting the film.coolbegin.com website. They've done nothing wrong at this point.

Doubtful, but it would be nice to get a reply from Yahoo regarding this.