How does Google index protected content sites?

[imulus]

I've done a number of scientific searches and whenever I cross a result from www.nature.com it blocks me with a request for a member id and password. If this area is protected how is Google able to index the content? I only ask because we would like to setup a website for a client in the same fashion as nature.com.

Any insights into this?

[dannysullivan]

Let's make it easy for everyone to understand. Search for site:nature.com on Google, and you get a list of all pages from that site. Notice that many of them have no cached versions. Notice also that some of them have different titles listed that what you'll see when you clickthrough to the page.

Basically, Google allows some sites to cloak/feed it content. I'm guessing this comes through the Google Scholar program, which is described more in these articles:

Google Scholar Offers Access To Academic Information

Working With Google Scholar -- And More Approved Cloaking

Ordinary web sites aren't allowed to do this. It generally been something they do for scholarly content and it seems some news sites. I'd contact the business development department and Google to see if they'll allow you to do it. Or maybe someday, Google will make this possible for any joe schmo web site to do. That would be nice.

[vicyankees]

It's one of 2 things:

Google Premium Content - tied to their sitemaps feature which allows sites to provide Google with a sitemap feed in which Google will index the first page provided the site allows any user the First Click to be free to view that content and any subsequent requests can then require login/registration.

Pure Cloaking - Allow the spiders to pass the registration wall based on their useragent.