Working with duplicate content

[I, Brian]

I'd like to open up discussion on the issue of duplicate content issues: when is it a problem, how to deal with it, and when to deal with it.

Now, the basic response with duplicate content issues is obviously to avoid duplication issues as best as possible.

However, sometimes it's hard to avoid - for example, the vBulletin software I'm posting on here the SEW forums will create multiple URLs for calling up this thread.

In these instances, Google at least tends to be fine in simply filtering out duplicate versions of the thread, so that only one will tend to be returned in searches, rather than a strong of exact results.

Now here's the crux - when is duplicated content an issue that has to be proactively dealt with by the webmaster, rather than search engines?

Meaning - is there a point when duplicated content start to trip off warning bells at search engine headquarters?

If there are possibile penalisation issues relating to duplicated content, then what would search engines generally regard as duplicate content?

For human users it's easy enough to recognise the same article posted in two very different websites, but what criteria do search engines use for tagging documents as duplicated?

[Chris Boggs]

we are working on this problem for a couple of clients. Apparently the "Bruce Clay method" found here is excellent and SEO-friendly. Any others' thoughts on the "IP-Funnel" method?

[KeywordMonkey]

Here's a situation to consider:

Site A adds content article X.

3 hrs later their sister site, site B, adds article X in their different site design.

Both sites push their "own" articles on their homepage and menus and have different site designs. They link to each other a lot. The sites grow independently of each other as well as having this "shared" content. They have separate brands offline - so not just carbon copies in a different content template.

One thought I have is "if they want to read article X send them from site B to site A" as that site has the brand which specialsies in that topic area.

But of course that will affect existings listings in SERPs and so individual site traffic.

So, what do the engines think? Do they see duplicate content? Or is making content from site A accesible to the audience of site B, who might otherwise not find it, OK this way? Has anybody encountered this before?

[I, Brian]

A quick initial read-through tells me the "IP funnel" method effectively involves using .htacces 301 redirects. However, my recent experience with 301 redirects is that there is no SEO benefit passed on so I'm pretty loathe to try that method.

That's why I'd like to get a handle on the finer niceties of what is and is not classed as duplicate content from a search engine point of view, and how to ensure that preferred pages get the ranking focus when it comes to duplicate filtering.

[KeywordMonkey]

Originally Posted by I, Brian

That's why I'd like to get a handle on the finer niceties of what is and is not classed as duplicate content from a search engine point of view, and how to ensure that preferred pages get the ranking focus when it comes to duplicate filtering.

I strongly second that - a sub-topic for the indexing discussion at SES NY maybe?

[I, Brian]

I figure this is an area where some SEOs will have a lot of experience.

Actually, after procrastinating for months I finally took out a SEW membership to read the full length of this new article here:
Avoiding Search Engine Woes with Multiple Domains and Websites

I figured maybe there would be a few anonymous examples showing how different people have approached the issues from different angles.

However, it doesn't actually say anything other than "best sites stay in". You can paraphrase it in a single sentence:

"Um, ah - yes, sometimes when you use multiple domain names and duplicate content you might run into issues".

A tad disappointed.

[Chris Boggs]

Originally Posted by I, Brian

A quick initial read-through tells me the "IP funnel" method effectively involves using .htacces 301 redirects. However, my recent experience with 301 redirects is that there is no SEO benefit passed on so I'm pretty loathe to try that method.

That's why I'd like to get a handle on the finer niceties of what is and is not classed as duplicate content from a search engine point of view, and how to ensure that preferred pages get the ranking focus when it comes to duplicate filtering.

Yeah wouldn't we all?

regarding the experience w/301 redirect you say there is no benefit passed, but has there necessarily been any lost?

Let’s try a specific example: Site A has about 6 URL’s that are being pointed to it. The IP Funnel would place another “layer” between the pointed “top” level URL’s and, according to the article/Bruce Clay, “will redirect to the main site so that any links and status is passed.” You say this is false. I would love some proof even if you could IM me with it…it seems that this method and claim is accepted by most. But you know what “they” say: never listen to “they,” because “they” can get you in trouble.

[I, Brian]

It's not so much that the article is false, as much as I think there have been sea changes in how Google handles redirects over the past year.

A year ago, you 301, you got PR and rankings attributed to the target site relatively quickly. I don't see that anymore - you just wipe out the site being redirected. And god forbid you get it slightly wrong and 302 your target site into oblivion as well. So I'm a little averse to using redirects at the moment.

[Mel]

I have recently used a 301 to a new URL and all the Pagerank and backlinks showed up in the new URL in 48 hours.

You might want to read Googles patent 6,658,423 regarding Detection of Duplicate and Near Duplicate Files

[I, Brian]

Mel, did you maintain prior rankings for your keywords?

EDIT: Good call on the patent - will favourite and read later.

[Mikkel deMib Svendsen]

The main reason for me to ALLWAYS handle ANY and ALL issues with duplicate content is very simple: I want to have control and not leave anything to the ever changing nature of search engines.

I want to focus all my forces on one spot - one adress for each unique page. I want all engines to index only that one and all users to bookmark, links and talk about just that one. I don't want to create ANY kind of confusion with users and engines alike.

[Mel]

Yes prior rankings for search terms were maintained and with the site being optimized after it was moved other search terms rankins are now increasing very nicely.

[I, Brian]

Apologies - I should have qualified that in terms of anchor text to the site. That's where I saw the big problem - target rankings using anchor text falling by the wayside once redirected, though no problems with internal anchor text.

[Chris Boggs]

Originally Posted by Mel

I have recently used a 301 to a new URL and all the Pagerank and backlinks showed up in the new URL in 48 hours.

You might want to read Googles patent 6,658,423 regarding Detection of Duplicate and Near Duplicate Files

wow! talk about headache-inducing text.

Any comment on the IP Funnel idea related to this? Could two sites have similar content yet be separated by the "feeder site" from the IP Funnel idea and thus not be penalized? Sort of like if a site has a org. and a .com they are not penalized? Am I mixing subjects too much here?