Hi

Have searched this site for a while but can find no answers to help me resolve something.

I run a site of subscriber-only content and I want to submit all that content onto google.

How can I allow google access to the content for indexing without compromising subscriber-only access? And how can I do with this without cloaking or redirects?

Cheers

Al

You cant.

What many people do though is to allow the bot AND the user to see the first 200-300 words, it's enough to get indexed, works nicely as a teaser, doesnt involve cloaking or redirects and keeps everybody happy :cool:

My company has encountered a similar problem, though not exactly. Basically we have content that we want the bots to reach, but we don't want to provide a list of that content in the form of links for reasons involving our competition. What we've done is to link to the content using clear pixels on our site, so that your average user won't see it or ever click there but the bots can still spider it. Our tactic is to do one clear pixel in the footer that links to a page of clear pixel links for the content. No one but the bots have gotten to it yet, and we haven't seen any penalties.

Noustie, that's a VERY dangerous tactic and I advise you to find a different way ASAP.

SE's hate that and will penalize it wherever they find it. It's just a matter of time.

Here is the problem, you guys. Lets say this works and you now have a fully indexed site that shows up for your keywords. Then what?

I type in a query and see a link to a page that might answer me. When I go there I get a "Subscribers Only" screen (or not authorized). I get REALLY annoyed. I report it as a bad or broken link. It gets removed. Or I go start using someone elses search engine that doesn't effectively make me sign up for something different every time I do a search.

The bottom line is that trying to tell google something and trying to avoid telling regular people that same thing is cloaking. By definition. Regardless of the tactic you use.

Take a look at this very website: www.searchenginewatch.com (http://www.searchenginewatch.com/). You will see that they have a bunch of free content, and also a bunch of partial articles that you have to log in to in order to read the rest. This is VERY effective (it's how I became a member a long time ago).

Bottom line, if you are trying to figure out how to show an SE one thing and a person something else, you are heading in the wrong direction. You can show the same thing to both and still encourage signups.

Ian

Yeah I agree with you mcanerin - problem is that we have a product that is very valuable and we have to charge for it and we cannot give away more than a handful of any articles at any given time.

Not enough to do us justice on any SE....I have no problem letting people know even at the SE results level that it is locked content but I cannot do that as far as I can tell as google ignores meta tags now.

I really think there is an awful lot of content out there that is locked and not being spidered...

Surely google could come up with a solution...

The only other way I have thought of thus far that is effective and legal is to use a script that says any user from the google bot specified IP range is a google bot and the system should recognise it as a subscriber etc.

That way google gets in and spiders...

This started by the way as we were approached by a SEO who would create a mirror of the site for google to index and then redirect and they claimed they could redirect 'legally' and they then charge a click through rate based on the redirects. But whilst we liked the idea of exposing our content to the crawlers as our content markets itself we remained unconvinced of their methods.

any other thoughts...

cheers

The only other way I have thought of thus far that is effective and legal is to use a script that says any user from the google bot specified IP range is a google bot and the system should recognise it as a subscriber etc.

And thats really the best way. You can buy a subscription to Ralph's (Fantomas (http://fantomaster.com/)) spider list and match the users IP against it to allow spiders in only then just use a no arcive tag to keep the meat of your content out of the cache.

If you go by user agent only then eventually it will bite you in the rear...

Will progress down the route and report back...