Time For An Indexing Summit?

[dannysullivan]

Bloggers continue to be upset over comment spam? Forum owners, those with guest books and others have dealt with the problem of link drops for ages. Web site owners have a host of other indexing problems and needs they'd like dealt with. The last time search engines came together to consider how to index the web in a coordinated fashion was 1996! Isn't it time for new developments to be considered? And what types of special tags or features would you like?

My blog post today (Comment Spam? How About An Ignore Tag? How About An Indexing Summit!) looks at the issue in more depth. Feel free to read if you want more background. But definitely please contribute here about what you'd like to see offered to site owners.

[I, Brian]

I still think the entire issue boils down to one single feature: publisher responsibility.

Search engines will simply index what is published - it is entirely up to webmasters to take responsibility for what they publish - and if there is a problem with the process then it is up to the software developers themselves to help make it easier for users on the issue of perceived spam.

Publicly publishing content of your own to the internet is one part of the responsibility - but when anybody creates opportunities for third-parties to upload their own material to be published on a site, then it is completely up to the site webmaster to continue to take responsibility for it on their site.

That means working actively and proactively.

I admin multiple forums - part of our remit for myself and forum staff is to provide a moderated community, with responsibility for what is published - that means removing obvious advertising, objectional material, and anything else in clear and persistent violation of whichever applicable TOU is in force. I accept responsibility for what I publish on my domains.

Blog software has been a big problem - firstly, because the blog developers never took it upon themselves to install key default safety features such as redirected URLs and disallowing HTML in comments.

Six Apart recently moved to try and reduce the load on their servers from comment spam - especially automated scripts - but they didn't really tackle the key issues of how to prevent the problem in the first place - just load issues.

Ad tags and the like I find to be an utter distraction, and simply a way in which software developers and end users can avoid taking proper responsibility for their own publishing.

[dannysullivan]

I still think the entire issue boils down to one single feature: publisher responsibility.

But how do you be responsible when some of the things you want to do might be viewed as spam.

I might want to be responsible by not "publishing" my site navigation or blog comments to a search engine. I can't do that. If it's on a page, I'm suppose to show the entire page to the search engine. If I strip portions of it, then I get accused of cloaking

Search engines have provided virtually nil input into helping content management systems and webmasters become more creative or sophisticated with what they deliver.

[greenleaves]

I might want to be responsible by not "publishing" my site navigation or blog comments to a search engine. I can't do that. If it's on a page, I'm suppose to show the entire page to the search engine. If I strip portions of it, then I get accused of cloaking

Search engines have provided virtually nil input into helping content management systems and webmasters become more creative or sophisticated with what they deliver.

The thing is, if the SEs allow people to publish certain parts of a page, and others not, then that oppens up a loop hole which spamers can take advantage of. The SEs are looking out for themselfs. They try to deliver the most relevant results in the most convenient manner (by convenient I mean for themselfs and for the end-user). They ow us nothing. Why would they go out of their way to help publishers... its in our interest (more then theirs) to be in their index. 6,000,000,001 or 6,000,000,000, pages indexed, whats the difference to them?

Publishers and SEs need each other, but it is the SE that has the upper hand, always will. It would be nice of them to give us more options, but I don't think they are going to be willing to spend the man-hours developing and researching something that, as far as I can see, would not bring them any direct benefit, only potential losses.

[Mikkel deMib Svendsen]

Danny, I certainly think such a summit would be great - I am just not sure the time is right. I am afraid that the general trust between the different participants in such an event is just not good enough at the moment. There are too many hidden agendas, general mistrust and bad behavour (on both sides). I think we need a better enviroment for sumit like that to actually come out with valuable results. But, I like the idea

[Chris Boggs]

Originally Posted by Mikkel deMib Svendsen

Danny, I certainly think such a summit would be great - I am just not sure the time is right. I am afraid that the general trust between the different participants in such an event is just not good enough at the moment. There are too many hidden agendas, general mistrust and bad behavour (on both sides). I think we need a better enviroment for sumit like that to actually come out with valuable results. But, I like the idea

Thanks again Mikel for a piercingly accurate description of the state of our industry. I too wonder who could be a part of this summit, other than perhaps Danny Sullivan who may be the only person in the world truly considered to be "impartial" about search engine rankings and companies within the industry.

I am of the opinion that mature and business-evolved people can be found to participate in such a summit. Perhaps it could be moderated by the new voice of Google: Leslie Stahl

[rustybrick]

I think there is more then one individual in this world that is impartial and unbiased in this regard.

I think the time is right and let Danny Sullivan pick those who should manage/lead the summit.

[Mikkel deMib Svendsen]

I don't think it has so much to do with being impartial - I don't think speakers or panelists of such a summit have to be that but they have to WANT the summit and WANT to truly share great ideas for the better of all. If we don't have the right degree of trust and honest will between the "brains" in the summit I am just not sure how well the outcome will be.

I very much like Noel McMichaels "Search Ecosystem" ideas but the realities of the industry right now are that some participants are pissing in the water others have to drink... so to speak

But having said that I would not hold anyone back that wanted to start such an event or ague against actually doing it. We just have to be realistic about the results. Who know, hopefully by time we WILL get a better environment than we have now - some might say, a more mature one

[I, Brian]

Originally Posted by dannysullivan

But how do you be responsible when some of the things you want to do might be viewed as spam.

That's why I mentioned about developers helping webmasters make those decisions. Someone may wish to admin a blog, but not have time to deal with the sometimes time-consuming task of removing comment spam - that's why software developers need to address these issues.

If all blog software came with:
redirected URLs
no HTML allowed
registered users only

and there was a campaign to move all users to the new versions, you could strangle the practice of comment spam within 18-24 months.

The question is really why have the developers not addressed the concerns as yet?

Even the recent moves by Six Apart to make comments moderated by default will not apply the vast number of Movable Type blogs already in operation.

Originally Posted by dannysullivan

I might want to be responsible by not "publishing" my site navigation or blog comments to a search engine. I can't do that. If it's on a page, I'm suppose to show the entire page to the search engine. If I strip portions of it, then I get accused of cloaking

Flash or Javascript your navigation, and make your comments area accessible for registered users only.

That's effectively addressed the spiders, but it hasn't at all addressed the practice.

Search engines have provided virtually nil input into helping content management systems and webmasters become more creative or sophisticated with what they deliver.

It's not the search engines who need to be there - just the software developers with an understanding of how and why their software distributions may be targeted for search engine purposes, and how they can help webmasters responsibly address such issues more easily.

Or am I off-topic now?

[KeywordMonkey]

Just found this - FyberSearch Fights Comment SPAM with a New HTML Tag

(Hat tip to where ever it was posted first, can't remember where).

[dannysullivan]

The thing is, if the SEs allow people to publish certain parts of a page, and others not, then that oppens up a loop hole which spamers can take advantage of.

Spammers already do this. They can use noscript tags, cloaking, noframes areas and so on. I'm sure they can abuse new techniques. But the problem is people who don't want to abuse things but want certain capabilities don't have them at all. Look back at my piece and about Brad Choate. Here's a guy who is cloaking but had no idea he was doing it -- and really, shouldn't even care. It's his content. Yep, we get traffic from search engines. But we should be able to push for changes that will really help content owners, as well.

I don't think they are going to be willing to spend the man-hours developing and researching something that, as far as I can see, would not bring them any direct benefit, only potential losses.

If only on the comment spam thing, Google will need to act or continue to be (unfairly) the whipping boy of all the bloggers upset over it. So they have a public relations interest to begin with. Secondarily, as more and more content spills into the web, they've got an economic reason to try and work more directly with publishers. And they are -- look at Yahoo with its new RSS enhancements, or Google with its print partnerships. What they aren't doing is talking to each other or to the publishers as a whole for any type of overriding stuff that might be needed.

You could strangle the practice of comment spam within 18-24 months.

And that helps the bloggers. Guest book people? People with other indexing problems? To be, the blogger complaints are the visible tip of the iceberg. We have publisher needs, and I think the search engines need to get more coordinated on some publisher solutions.

Tell you what, thinking about it more, I'm almost certainly going to do a session on this at our next show. I'll try to get some reps together on a panel, then let the audience throw out what they want and see if we can't at least discuss some of the problems and practicalities of what's raised. I'm not foolish enough to assume that we'll see immediate changes -- but I'd like to at least see some ideas put out there and be discussed.

[I, Brian]

Originally Posted by dannysullivan

Tell you what, thinking about it more, I'm almost certainly going to do a session on this at our next show. I'll try to get some reps together on a panel, then let the audience throw out what they want and see if we can't at least discuss some of the problems and practicalities of what's raised. I'm not foolish enough to assume that we'll see immediate changes -- but I'd like to at least see some ideas put out there and be discussed.

If you do, it would be great to get software development companies up there - holding search engines as responsible to deal with spam issues could be dangerously close to asking them to censor parts of the net - a far more dangerous issue, IMO - better to empower webmasters to responsible "voting" via linkpop, than asking for their votes to be completely discounted.

2 opinionated c.

[orion]

I found the following as great ideas.

1. an indexing summit

2. a special tag for no indexing specific links in a document as mentioned in the blog.

My comments:

#1 is much needed.

About #2, I look at the W3C specifications but could not find anything for this type of specific tags. I will check a bit deeper later. Meanwhile, it occurs to me a possible solution.

As part of some old HTML DOM and SERPS experiments, back in 1998 I played with making links illegal and checking the effects on SERPS. One way to do this is using nested anchor tags. IE renders the inner link active but still are illegal, according to the W3C specification. See Nested links are illegal.

For instance, the following links are illegal. To render the urls, I removed the prefix part, but you get the idea (the http prefix and www part should be used in a real example)

<a href='****://***.google.com/'><a href='****://***.msn.com/'>MSN</a></a>

<a href=' '><a href='****://***.msn.com/'>MSN</a></a>

My data is old, so I'm not sure which modern engines are ignoring nested links. Still the inner link (MSN) is active and clickable in IE. (I don't know about other browsers).

Just a pausible solution to the problem of links placed in forums we don't want to be indexed by search engines.

However, I cannot assure if this idea could/could not be used for spamming, too. and due to lack of time, I don't have current data to argue one way or the other.


Orion

[mcanerin]

One issue is that there has to be something in it for the software people to want to do it. Generally, the push in software is to add more functionality and options, not less.

First, it gives a competitive edge selling to people who often buy the package with the biggest laundry list of abilities because they are not sure what they want yet, and second, it keeps the programmers employed...

Ian

[Mikkel deMib Svendsen]

There are plenty of ways to hide a link from being indexed - I don't think thats the issue. The issue is that the methods available could all suffer from editorial or algoritmic penalties. What we need (if we need it) is a recognised tag that we know we can use and recomend clients to use without the fear of getting penalised for the use of it.

Most of the enterprise search solutions (on-site search) I've seen and used have such a tag available. This is the way they most often exclude navigation or other site-wide information that dosn't contribute to the direct understanding of each page.

[telNform]

An Indexing Summit is a great idea ... I would love to help out.

telNform

[greenleaves]

Tell you what, thinking about it more, I'm almost certainly going to do a session on this at our next show. I'll try to get some reps together on a panel, then let the audience throw out what they want and see if we can't at least discuss some of the problems and practicalities of what's raised. I'm not foolish enough to assume that we'll see immediate changes -- but I'd like to at least see some ideas put out there and be discussed.

I will be glad to assist. And if any of the topics brought up during the event cause a change (for the better) for publishers, I'll be twice as glad. I guess I just can't help being cynical.

[orion]

Originally Posted by Mikkel deMib Svendsen

There are plenty of ways to hide a link from being indexed - I don't think thats the issue.

1. I don't think is about hiding links but about selective indexing according to editorial guidelines and instructing an engine not to indexing a visible link.

2. Still I conceed such summit could expose even more broader and important issues than mere tag instructions.

3. Timing. And I do feel the time for that kind of summit is right, but need to be carefully crafted. The question is how many would be inclined to attend/contribute to it. I know I'm more than inclined to attending it.


This post is given in the spirit of Danny's invitation.

Originally Posted by dannysullivan

My blog post today (Comment Spam? How About An Ignore Tag? How About An Indexing Summit!) looks at the issue in more depth. Feel free to read if you want more background. But definitely please contribute here about what you'd like to see offered to site owners.

Orion

[figment88]

I'm not so sure an indexing summit is such a good idea. I think the main problem with SPAM of all types is that the engines already work too much alike.

Back in the olden days when none of the engines worked very well, each at least had very different indexing rules and ranking algorithms. A site could do great on one search engine and be nowhere on all the others.

Now all the engines have pretty much the same secret sauce. Google has toned down the importance of links, but still all major engines favor them leading to all the the types of link spam mentioned. Well if there was a major engine or engines that didn't care about links there would be less incentive to link spam.

Search engines should look at things like comment spam as golden opportunities to differentiate themselves from the pack.

[I, Brian]

Originally Posted by mcanerin

One issue is that there has to be something in it for the software people to want to do it. Generally, the push in software is to add more functionality and options, not less.

First, it gives a competitive edge selling to people who often buy the package with the biggest laundry list of abilities because they are not sure what they want yet, and second, it keeps the programmers employed...

Ian

When vBulletin were informed of the "memberlist spamming" technique, they very quickly implemented a solution where vBulletin admins could automatically close the memberlist or else only show members who had made x posts. Both empowered the webmaster to combat "memberlist spamming" according to different requirements.