Google Hijacks Batman Room Decor Listing At MSN!

[Marcia]

See for yourself

http://search.msn.com/results.aspx?q...cor&FORM=QBRE3

And here's a screenshot in case their rankings drop

Google has Batman decor?

[Robert_Charlton]

Marcia - Who said PageRank was dead?

Google's now dropped to the middle of page 2, though. Maybe Bill heard about it. They'd better work on their onpage optimization a bit, or they're going to lose on this one.

[martinibuster]

There is one site in those serps that has three pages in the serps, one of them their Yahoo store domain. This site coincidentally is the only one in the MSN serps that also ranks in Google for those same terms in the top ten.

I did a google search for the term google on that site and this one site contained those words in the results page.

There's some kind of JS with the words google in them on those pages. When you check the cache you see the words Google Trout on them- they're in supplemental.

Not sure what any of this has to do with the MSN result, but the code is so bad it made the W3C validator puke. Could this have tripped up the MSN spider?

Maybe has nothing to do with it, but an interesting coincidence.

[Marcia]

Oh, it sure does have something to do with it, MB. There's a bug, MSN is having a problem with crawling Yahoo stores, apparently.

Actually, Google was having a problem with crawling Yahoo stores right around the Florida update with all kinds of duplications turning up in the SERPs because of the domain ranking and the Yahoo subdirectory URL ranking for the same thing - on the same pages. Also, they were indenting different Yahoo stores with each other if they had the same search terms. One Yahoo store wasn't being differentiated from another. The Google SERPs were literally *flooded* with Yahoo stores at that time, right after Florida.

The problem (I think, as best I can remember that far back) was in Googlebot crawling in to the Yahoo stores' URLs from Yahoo! shopping as well as to the domain from links. I wrote Google with multiple examples for about 4 days running - saying specifically that it was NOT a spam report, but a problem on their end. Finally, they responded by email (probably to shut the woman up from nagging them), saying it had been sent over to engineering. It was rectified very quickly and stopped happening.

It isn't an outing about anything to do with the site itself - it's a problem on MSN's end (and actually with what Yahoo store URLs do with crawlers) so it's OK to post this to point out what's happening with MSN:

The domain name, with an interior page indented right underneath - is ranking at #2 and #3 at MSN for that search term

http://www.storybooklane.com/

Google cache, with a cache date of Sept. 28th 02:46:28

http://216.239.63.104/search?sourcei...klane. com%2F

Here's the Yahoo store subdirectory URL, where the site actually physically resides - same page with the other URL ranking at #10 for the same search term on the same page

http://store.yahoo.com/storybooklane/

Google cache of that URL dated Sept. 28th 02:46:28

http://216.239.63.104/search?sourcei...orybooklane%2F

Two URLs, both showing toolbar PR4, which is incidental to this issue. But apparently MSN's bot is crawling both ways, hence the duplications and the same page showing with two different URLs.

This is what the domain name shows for having the word google on the site

http://www.google.com/search?sourcei...e%2Ecom+google

The googletrout was just a clearance item linked to from that Supplemental clearance page on the site. And here's what turns up at Google for having the word google on the site with a search on the URL for the Yahoo subdirectory site where it lives

http://www.google.com/search?sourcei...o%2Ecom+google

All those pages from shop.store.yahoo.com - Yahoo stores can now be 301'd to the domain names if they have them, which obviously not all are, but I dont think that 100% deals with the issue of shop.store.yahoo.com

Now it needs more checking out how all that is looking in MSN's index with those different URLs for Yahoo stores.

[DaveN]

OK I think it's msn trying to fix there 302 problems .. lol

http://cc.msnscache.com/cache.aspx?q...-US&FORM=CVRE8

Headers for http://groups.google.co.uk/froogle_u...4AAAAAAAAAA A (64.233.167.147):

HTTP/1.1 302 Found

Just a guess ... but why do google still use 302 ..

DaveN

[DaveN]

see what I mean

http://www.google.co.uk/search?q=Bat...en-GB:official
http://search.msn.co.uk/results.aspx...ted+B ean+Bag

DaveN

[Marcia]

I've seen those Google groups URLs all over the place, besides still seeing affiliate links turning up in search results instead of URLs of pages. I think that Google groups 302 thing is a different issue from Yahoo shopping though, with seeing the dups turning up, so it looks like more than just one problem there.

The title here may seem "cute" but imho these are some fairly serious issues.

[dannysullivan]

For the less savvy who may not be figuring out what's happening, working with Dave, I've done a blow-by-blow over here: Listings Hijacked At MSN, With A Little Help From Google. Short story? Google doing temporary redirections which is causing MSN to let it hijack the listings of other sites.

[Alan Perkins]

Originally Posted by dannysullivan

I've done a blow-by-blow over here: Listings Hijacked At MSN, With A Little Help From Google

From that article...

Hey! What's MSN doing crawling Froogle anyway? The robots.txt file there should be keeping it out, right? Sure. But if some site has made copies of Froogle results, scraped the content as fodder for a fake blog or something else to attract traffic, MSN might crawl that and thus see the Froogle redirections.

That's not right. The only way the content could be associated with the indexed URL is if the indexed URL has been fetched in order to see the 302. And the robots.txt linked to by the article, http://froogle.google.co.uk/robots.txt, does "Disallow: /froogle_ ", so if the article was correct then MSN would have to be contravening robots.txt.

MSN is not contravening robots.txt though, because the robots.txt file that's in operation is not http://froogle.google.co.uk/robots.txt but http://groups.google.co.uk/robots.txt, which looks like this:

User-Agent: *
Disallow: /search
Disallow: /advanced_group_search
Disallow: /advanced_search
Disallow: /groups
Disallow: /group
Disallow: /tmpl
Disallow: /img
Disallow: /static

That file does not "Disallow: /froogle_", so MSNBot could legitimately fetch the URL in question.

[Marcia]

This gets better every day - now Google is ranking #1 for motion activated moose

http://search.msn.com/results.aspx?q...oose&FORM=QBRE

At least they're not for animated singing moose, but there are a few interesting sites turning up for that, who knows how (one of which redirects right away). Also included there is an example of an affiliate link as the URL in the listing with the actual site's title and description.

http://search.msn.com/results.aspx?q...oose&FORM=QBRE

Others have shown up with the affiliate link as both the URL and title with no description - quite possibly no tricks involved and not deliberate on the part of the origin on the link. I'm quite familiar with one particular site that's happened to quite accidentally, not a clue until it was seen at MSN with the affiliate ID for the title and URL.

Those must depend on how the links are set up and through which network they're run, since there's an intermediary "stopover" with the network links for tracking before getting to the final destination, so it's not unrelated to redirection issues.

So this is about it for the 3 "bugs" I've been noticing.

[eileen kowalski]

This isn't the first time that Google has done this kind of hijacking with 302s in MSN. Remember when GoogleGuy started asking for examples of 302 hijacks in late November/early December? Considering that they knew it was an issue in their own engine, I was a little disappointed that they chose to use 302 redirected URLs to link to other site's review pages when they added aggregated review content to Froogle in mid-December. Thankfully, that instance of 302 hijacking was quickly resolved.

How it's happening currently: Go to http://groups.google.co.uk and click on the "More" link in the top Google navigation. You should end up on http://groups.google.co.uk/intl/en/options/. Then, click on the Froogle link. Notice how this URL displays a duplicate of Froogle underneath it: http://groups.google.co.uk/froogle?hl=en. Hopefully this is just a bug since the other similarly configured links (i.e. http://groups.google.co.uk/blogsearch?hl=en) are returning 404 Page Not Found.

To see the full damage in MSN:
http://search.msn.com/results.aspx?q..._url&FORM=QBRE
http://search.msn.com/results.aspx?q..._url&FORM=QBRE

I'm not sure how the groups.google.com URLs got indexed. Maybe they used to have a similarly configured "More" page on groups.google.com at some point that they have since fixed?

For the sake of merchants listed in Froogle, I really hope all of this gets resolved quickly. Christmas is right around the corner. Last thing you want is Google hurting your rankings in MSN.

[Robert_Charlton]

Originally Posted by eileen kowalski

This isn't the first time that Google has done this kind of hijacking with 302s in MSN.

I think that the term "hijacking" in this context continues to be unfortunate. This is not about purposeful hijacking. It's for the most part about the bugs involved in indexing temporary redirects, and about all of the engines being tardy in recognizing that they had some problems in this area.

[Alan Perkins]

Originally Posted by eileen kowalski

I was a little disappointed that they chose to use 302 redirected URLs to link to other site's review pages when they added aggregated review content to Froogle in mid-December.

Like any Webmaster, it's their choice whether they use a 301, a 302, a 307 or whatever. Their choice should not be dependent on a search engine's inadequacies, it should be dependent on which type of redirect is most appropriate (temporary or permanent). It's not Google's problem that MSN can't handle a redirect correctly. It's MSN that is in error and MSN that should change.

For the sake of merchants listed in Froogle, I really hope all of this gets resolved quickly. Christmas is right around the corner. Last thing you want is Google hurting your rankings in MSN.

The pages are indexed and when a listing is clicked, the searcher ends up at the corect page on the merchant's site (via a Google redirect). The main damage is not to rankings but to branding, since the wrong URL is listed in the SERPs. This may negatively influence CTR. OTOH, the appearance of the Google URL may positively influence CTR. Either way, the problem is MSN's - not Google's.

[eileen kowalski]

The real question here is: have merchants seen a drop in their MSN revenue, traffic and indexed pages as a result of their pages also being indexed under the Google URLs? IMHO, this is what constitutes a hijack (whether it was purposeful or not). Any anecdotal evidence?

I agree that MSN should fix their search engine, but I also think that Google could have decided to implement their outbound links differently knowing that 302 redirected URLs were in issue in their own engine.

Really, this all speaks to a larger issue about redirection and referrer tracking. If there was a standardized way to specify the referrer in the URL and all search engines stripped this information from the URL during the indexing process, then people could reserve 301 redirects and 302 redirects to indicate when a URL has actually permanently or temporarily moved, right? Or maybe the server can issue another server response to indicate that the URL is a tracking URL? Or maybe we can even add an additional hsend= to the link protocol? As the web becomes a more commercial entity, I really think a universal response to referrer tracking would solve issues with indexing, accounting and more. Don't mean to get this too off track, but the beauty of the web is that we should be able to fix things like this on a large scale, especially if companies like Google and MSN and (dare I say) experienced webmasters and SEOs put their heads together.

[Marcia]

Like any Webmaster, it's their choice whether they use a 301, a 302, a 307 or whatever.

Alan, Google is not "any webmaster," not by a long shot.

Their choice should not be dependent on a search engine's inadequacies, it should be dependent on which type of redirect is most appropriate (temporary or permanent). It's not Google's problem that MSN can't handle a redirect correctly. It's MSN that is in error and MSN that should change.

Sorry Alan, but Google has had their own share of inadequacies where 302's are concerned. I think the point being made is that they should know better.

Come on, Google - fix it!

Have you or any or your clients had site/pages hijacked by 302's? There are no doubt different perspectives on the issue, depending on whether one has been hurt by it or not - or whether or not their clients have been blown off the face of the map by mishandling of redirects - Google included. Have you yourself or any of your clients personally experienced it? It may look quite different from that viewpoint, so the reply can possibly give some objective, non-rhetorical perspective and give us insight that's more in depth related to the core issues involved.

[Alan Perkins]

Originally Posted by Marcia

Alan, Google is not "any webmaster," not by a long shot.

Are you saying that's it's not Google's choice whether they use a 301, a 302 or a 307?

Sorry Alan, but Google has had their own share of inadequacies where 302's are concerned. I think the point being made is that they should know better.

They do know better, IMO.

Have you or any or your clients had site/pages hijacked by 302's?

I've been helping people to deal with this issue for many years. When it first cropped up, I diagnosed and reported the problem to Google and they fixed it. Over the intervening years they've broken and fixed it several times.

objective, non-rhetorical perspective

An objective perspective is exactly what I'm trying to give you - a perspective that doesn't remove all choice from a Webmaster and subjugate all control to search engines.

This is Microsoft's problem to fix now. The problem is being highlighted through Froogle, but there must be many many more examples of it throughout the Web. Rather than every one of those sites having to change to a 301 even if a 302 was more appropriate, should not Microsoft fix their flawed algorithm?

Originally Posted by eileen kowalski

If there was a standardized way to specify the referrer in the URL and all search engines stripped this information from the URL during the indexing process, then people could reserve 301 redirects and 302 redirects to indicate when a URL has actually permanently or temporarily moved, right?

Right!

I described how I thought it should be handled in this post and this post.

[dannysullivan]

MSN is not contravening robots.txt though, because the robots.txt file that's in operation is not http://froogle.google.co.uk/robots.txt but http://groups.google.co.uk/robots.txt

Yep, I agree. I'll have to correct that.

I think that the term "hijacking" in this context continues to be unfortunate. This is not about purposeful hijacking. It's for the most part about the bugs involved in indexing temporary redirects, and about all of the engines being tardy in recognizing that they had some problems in this area.

I guess it doesn't matter to me whether the intend was on purpose or not. The result is the same. The listing of another site has been hijacked. The actual URL is Google's control. That wasn't Google's intent, and it is largely MSN's fault it is happening, but I think hijacking helps underscore what a serious problem it is.

Like any Webmaster, it's their choice whether they use a 301, a 302, a 307 or whatever. Their choice should not be dependent on a search engine's inadequacies, it should be dependent on which type of redirect is most appropriate (temporary or permanent). It's not Google's problem that MSN can't handle a redirect correctly. It's MSN that is in error and MSN that should change.

I believe Google itself has advised others doing redirection like this to do 301s rather than 302s, so while I agree it is MSN's problem, Google takes a hit for not following the advice it has recommended to webmasters, if that's indeed what they've said.

[Marcia]

First, I'd like to thank Eileen for some great posts here in this thread; she's got the issues down just right, and well spoken, too! It's all about how the situation affects merchants, in the long term, and looking further - how do the search results impact users of the search engine, which IMHO is the bottom line.

While there's been at times an adversarial relationship between search engines and marketers, IMHO the bottom line is that the quality of search results impacts both groups equally, and differences put aside, the quality and accuracy of search is vitally important to both "groups." So essentially, in spite of any possible adversarial issues, there's a partnership aspect that benefits both.

For MSN Search, increase of market share and maintenance of their user base for search are important, and those same things are equally important to us as marketers. For those of us whose demographic is well represented at MSN, it's critical that the quality of search results be accurate and relevant, and provide quality search results for those users so they keep coming back. That really can't be expressed strongly enough.

There's a symbiotic relationship between us that relies on accuracy and relevancy on both sides. Sites listed in search should be for the URLs belonging to those whose sites are relevant for the particular searches, with the description snippets, page titles and URLs being properly accurate and descriptive. That isn't the case with "hijacked" domains and URLs, so thinking long term and for the better good, resolution of the issue is ultimately to everyone's advantage.

[Alan Perkins]

Originally Posted by dannysullivan

I believe Google itself has advised others doing redirection like this to do 301s rather than 302s

Is Google's advice actually documented anywhere? All I've seen are odd snippets that (in a nutshell) advised people to use a 301 (Permanent Redirect) where a 301 was appropriate, and a 302 (Temporary Redirect) where a 302 was appropriate.

The problem is that people have trouble working out which of a 301 or 302 is appropriate in a given instance. Often people don't really care, and just treat the two interchangeably (they need a redirect, they use "Redirect" in their httpd.conf file, it gets the job done, problem solved ...). IMO search engines have been far too literal in their interpretation of 301s and 302s, and that includes the current Yahoo interpretation.

so while I agree it is MSN's problem, Google takes a hit for not following the advice it has recommended to webmasters, if that's indeed what they've said.

I'm not so sure that they have given any specific advice. They've just been like any other Webmaster and not put too much thought into whether a 301 or 302 was most appropriate. In this case I'd have said a 301 was probably a better choice than a 302, so Google takes a hit for using a 302 when a 301 would have been a better choice.

But Webmasters generally still have to be able to use a 302 for redirecting to external URLs. We can't just make that "illegal" because search engines continue to mess up with it.

[Chris_D]

If you've changed your URL, or plan to, and would like Google to display your new URL, please keep in mind that we can't manually change your listed address in our search results. That said, there are steps you can take to make sure your transition is smooth.

If your old URLs redirect to your new site using HTTP 301 (permanent) redirects, our crawler will discover the new URLs. For more information about 301 HTTP redirects, please see http://www.ietf.org/rfc/rfc2616.txt.

http://www.google.com/webmasters/3.html