MODERATOR NOTE: About.com discussion split from BlueFind Dir PR Drop?? (http://forums.searchenginewatch.com/showthread.php?t=3575).

One frustrating thing. About.com is very much pushing the limits on ethical SEO. They have sitemap pages and A-Z pages obviously intended for crawlers only. Called http://spiderbites.about.com/sitemap.htm. If they weren't owned by Google, would they get a penalty too?

I think many of the pages were not indexed and just recently were reindexed.

>About.com is very much pushing the limits on ethical SEO. They have sitemap pages and A-Z pages obviously intended for crawlers only.

ethics is really a bit of a hokie word as it relates to their seo techniques.

>If they weren't owned by Google, would they get a penalty too?

I do not think they are owned by Google.

My mistake, Google bought sprinks owned by primedia which owns About.com so there is a very close relationship and in my opinion, some of Google's largest distribution partners get a bit of a pass on things that may cause others to get dumped. Or at least I think that they are told how to push the limits in private. Only an opinion.

Google Buys Sprinks
Meanwhile, Google has acquired Sprinks, a paid listings service that was owned by PRIMEDIA. As part of the deal, Google will provide paid search and contextual listings to PRIMEDIA. The main distribution point for Sprinks listings has been on PRIMEDIA-owned About.com.

Doesn't matter anyway. I just felt bad that Bluefind looks like a penalty, maybe it's not after all. I read John Scotts forum and I am not sure anything has been resolved. I think we will know within a week or so if Bluefind returns to the Index.

About.com is very much pushing the limits on ethical SEO. They have sitemap pages and A-Z pages obviously intended for crawlers only.
There's nothing wrong with having a sitemap designed to help crawlers, assuming you haven't cloaked them, that is. The person who oversees site optimization for About speaks at our conferences all the time on the issue of making sure big sites can be seen by crawlers. Getting a sitemap that points at real content that humans can access is part of that. Search engine reps at our conferences say this, as does Google here (http://www.google.com/webmasters/guidelines.html):

Offer a site map to your users with links that point to the important parts of your site. If the site map is larger than 100 or so links, you may want to break the site map into separate pages.

Looking at the page you mentioned, I went over to the About.com home page. At the bottom is a completely visible link that says "Site Map" that leads to these pages. So they work for spiders, but they are also intended for humans. Difficult for me to see anything wrong here. Heck, it's far more visible than Google's own site map (http://www.google.com/sitemap.html).

The Google sitemap is certainly more attractive looking than the About.com map, but the About.com map is far more accessible. Google doesn't link to its sitemap from its home page as About does. The "more" link from the home page doesn't bring it up, nor does the About Google link. I have to drill into the contact page (http://www.google.com/contact/index.html) at Google to finally find it at the bottom of that page.

If there's content pages that are specifically created just for spiders, that's a different matter. Now you're talking low quality doorway pages. But exploring the links you posted, I don't see that. I see a sitemap that leads to pages that are already naturally part of the About.com site and other Primedia properties.

About.com might also do paid inclusion with Yahoo, so some things they provide may also be designed for that.

If there's content pages that are specifically created just for spiders, that's a different matter. Now you're talking low quality doorway pages. But exploring the links you posted, I don't see that. I see a sitemap that leads to pages that are already naturally part of the About.com site and other Primedia properties.
I believe that the issue is that a sitemap is expected to lead to pages on the site/domain itself where it resides - NOT other sites and/or other domains.

Site maps leading to other properties of a company could be construed as being doorway pages created for crawlers, so that is where the issue lies with the pages that link to all the pages on sites/domains other than the one where the sitemap resides.

For all practical purposes, it appears to be that some of those sitemap pages are created strictly for the benefit of spiders, with the intention of gaming Google for passing on PR to their other sites.

If we view it in the same way we would view it were it some other site doing it, like some Mom 'n Pop site, that's what it would look like. If it were others doing it, in those cases we'd not be so very kind as a general rule.

The question remains, a perfectly viable one: Iis that a viable technque for others to emulate if they've got a network of sites of their own, or does it take reaching a level of immunity to get by with it?

Just a newby here, but I respectfully disagree with you Danny.

Looking at the page you mentioned, I went over to the About.com home page. At the bottom is a completely visible link that says "Site Map" that leads to these pages. "So they work for spiders, but they are also intended for humans."

This page was not made for humans:

http://spiderbites.about.com/pirt_customrodderweb.htm

Obviously, Google is vague in their description below, but I think the above page is pushing the limits especially for "or setting up pages/links with the sole purpose of fooling search engines" Fooling is definitely a term for interpretation. Fooling them into thinking a page has link popularity?

However, certain actions such as cloaking, writing text that can be seen by search engines but not by users, or setting up pages/links with the sole purpose of fooling search engines may result in permanent removal from our index.

I absolutely would not create a sitemap like About.com for fear that I would get a penalty. Maybe you would and maybe you would be right. I am just a little guy and I am very cautious. I do agree that human friendly site maps are useful and I do use them, just not like the one in the link above.

Just a newby here, but I respectfully disagree with you Danny.


This page was not made for humans:

***//spiderbites.about.com/pirt_customrodderweb.htm

I'm inclined to agree with the poster. The link listed at this page and right before the last link sends users to

***//***.customrodderweb.com/offers.html

This other page has two links that redirects, one sending users to this forum

***//***.streetrodding.com/sr/index.cfm?fuseaction=forum.topics

Great! Evidently this is a well organized link distribution structure, not necessarily meant for humans.

I haven't check the rest of the links. The redirection url mechanisms are also interesting. I do see a lot of unfair things in the picture.

Orion

So are we outing About.com now? What does About.com have to do with Bluefind? Eh? :confused:

So are we outing About.com now? What does About.com have to do with Bluefind? Eh? :confused:

ya LOL - one of those analogies that basically bitch-slapped BlueFind off the page! ;)

Maybe a mod can split the thread?

Site maps leading to other properties of a company could be construed as being doorway pages created for crawlers, so that is where the issue lies with the pages that link to all the pages on sites/domains other than the one where the sitemap resides.


Well put.

I just visited the link

****//***.customrodderweb.com/offers.html

and now it sends me to a list of links pointing to way dissimilar sites offering different services. Here is a truncated sample with some text and urls removed.

Free Online Classifieds
Place Unlimited Free Photo Ads....

Place 50 Million Ads Free
1 Push Button Ease. No Limit

Get 10,000 free leads now
+1000 free hits

Make $**** Weekly Online
Processing Emails At Home Fun!

Local Dating & Personals
Find Fun, Dates, Love and More...

and the list goes on...


This gives me a flash back of the EZ searches threads and Google in many discussion forums, not that long ago.

Bottom line, to me this is not any different from what others do in the background (or in a corporate setting) for a living. Are we talking about double standards? Give me a break.


Orion

Per some of the posts above, I've split the thread. Also, we do have a no spam reporting policy on the board. However, we make exceptions when large, prominent sites are involved. About.com is one of those, which is why this discussion has been allowed.

Now as to whether About.com is spamming with its site map. First, I'll see if I can get the person in charge of About.com and Primedia's optimization to drop by and give you a first hand account. What I can tell you is that he knows and sees reps from Google and Yahoo all the time. I'd be incredibly surprised if About.com is doing anything they don't feel is OK.

Now does that also mean About.com might be getting special treatment, allowed to do things others aren't? Honestly, I doubt it. Google and Yahoo talk with people who operate major sites all the time for better advice on how to index the content that they want to get. For all I know, About.com itself may have been advised by the engines to do what it is doing. Or not -- again, I'll see if someone will come by.

I believe that the issue is that a sitemap is expected to lead to pages on the site/domain itself where it resides - NOT other sites and/or other domains.

So when I go to Google UK and view its site map (http://www.google.co.uk/sitemap.html), and I get links to Google.com in some places, that's not kosher? Or if I make a sitemap for SEW and link to our forums and blog, which are on subdomains, that's not allowed? Or if I go to Quicken's sitemap (http://quicken.intuit.com/commerce/catalog/fragments/qcom/site_map.jhtml), they're doing something wrong by linking to Quicken Card, Pocket Quicken, Quicken Credit reports or TurboTax? Hey -- it's your site. You can define whatever you want on a site map. If you think it makes sense to be there, put it there. And if you are running a large site or a network of sites, I think you will consider your site map differently than a small site contained in a single domain.

For all practical purposes, it appears to be that some of those sitemap pages are created strictly for the benefit of spiders, with the intention of gaming Google for passing on PR to their other sites.
That's a pretty strong statement. If you were trying to game Google this way, wouldn't it make a little more sense to at least get some decent anchor text into the links on the sitemap? If I go into the subpage (http://spiderbites.about.com/history1900s.htm) for 20th Century history, I see none of that. To me, it looks like they are using an automated script to break up the links on the site, which doubles up as a way for people who want a map to get one and also as a way for spiders to locate all the pages within the site -- which I'll say again, to me is perfectly fine.

This page was not made for humans

I absolutely would not create a sitemap like About.com for fear that I would get a penalty. Maybe you would and maybe you would be right. I am just a little guy and I am very cautious. I do agree that human friendly site maps are useful and I do use them, just not like the one in the link above.

I think you'd be OK if you wanted to create an automated listing of all your pages like that. Come back to the main point. You've linked to it from your home page. You aren't hiding it, and you're simply listing all the pages in your entire web site. You should feel perfectly fine doing that. Anyone should, honestly. It would be far, far better if the About.com page had more structure to it. Of course, if they did have nice anchor text, then we'd be accusing them of trying to get link boosts :) OK, seriously, the site map could be much better for humans visibilitywise. My guess would be simply that they don't see it worth putting more into the automation to do this -- and just making a map of thousands of pages is a challenge. It comes back to big site, different issues and tactics than small sites. Not necessarily double-standards.

I just visited the link
****//***.customrodderweb.com/offers.html
and now it sends me to a list of links pointing to way dissimilar sites offering different services. Here is a truncated sample with some text and urls removed.
So I bet what's happening here is that the Custom Rodder site, which I assume is one of Primedia's magazines, carries Google AdSense ads like you see at the bottom of the site's home page: http://www.customrodderweb.com/

Now I'd also bet that what happens is that the offers.html page is a file used for server side include into other pages. I'm especially betting this is the case becasue I see no title tag on it.

If you use an automated script to link to any HTML files in a directory, and you have included in there, then that page will get picked up. Probably what happens here. And the reason those Google AdSense ads then seem poorly targeted is because Google may simply not have indexed that "page" so doesn't know what it is about.

A further bit on this. Look at the left-hand nav of the site, and you'll see there's an Auto Offers section. Now click on links there, and you get keywords sent along with the offers URL to generate some context-sensitive ads: http://www.customrodderweb.com/offers.html?kw=buick+car+parts&kw_display=Buick+Car+Parts.

I suppose About.com might want to work to prevent pages like that from possibly being indexed -- but then again, Google also doesn't seem to be indexing them anyway.

Overall, what doesn't go away is that if Google has ad content partnerships with sites -- as it does with About -- then it will always have these type of questions lurking. Are you favoring a particular site, because you have a deal. Do About, Amazon and others get a boost. So far, they continue to say it won't have an impact and there is no favoritism allowed. But it leaves them in a weak position.

Thanks, Danny.

One more little thingy about the
****//***.customrodderweb.com/offers.html url. Today I look at it and is showing different link offers from yesterday. Clearly dynamically written and rotated as it can be seen from its source.

I don't understand why some claim this "site" "map" is for both human or spider consumption.


Orion

Today I look at it and is showing different link offers from yesterday. Clearly dynamically written and rotated as it can be seen from its source.
That content I'm pretty sure is being delivered dynamically via JavaScript from Google's ad syndication servers. So the rotation and targeting is largely a Google thing.

I don't understand why some claim this "site" "map" is for both human or spider consumption.
Well, I guess that's just me making that claim so far :)

I say it is for both because they link to it in visible fashion from the home page. If it wasn't intended for humans at all, they wouldn't have that link there. It could be much, much better for humans -- that's obvious. It looks primarily intended as a way for spiders to see what's on the site.

Even if it was a site map purely for spiders, however, I still don't see that as a spam problem. I wouldn't classify a site map for spiders as content specifically aimed at spiders. The latter, to me, is the idea of making doorways or nonesensical copy that you hope will reverse engineer a search algorithm. A site map for spiders, that's just a pure indexing aid. It's not misleading in my book to do that, to link to real content you have within your site. I'd honestly be very, very surprised if any major search engine had a problem with that.

Well, I guess that's just me making that claim so far :)


1. I agree with that.

2. Yes, Includes and JavaScript are used.

I guess, it would be OK if others start massively replicating About.com's link scheme and to their heart needs as far as all parties involved will be happy and OK with it.


Orion

it's wide spread too

http://www.google.com/search?&q=Primedia+%22For+Search+Partners%22

DaveN