For Reviewers and Submitters

[orion]

For Reviewers and Submitters

I’m starting this thread with one objective in mind: to serve as a reference point for

Beta test reviewers ; i.e., members interested in providing honest feedback
Submitters; i.e., authors and developers seeking honest feedback

Why do we need this thread? Simply put, because reviewing SEO/SEM/search-related software through a forum is similar to shooting at a moving target.

Consider this.

Products or services to be reviewed

1. could have merits or be a mere promotional exercise (e.g., drive-by spam)
2. could be unique or a derivative of something already available
3. could involve new technology or nothing new at all
4. can be legal or infringe on other rights and intellectual property
5. could be useful to SEO/SEM specialists or could be useless

Submitters

1. may be looking for honest feedback or don’t care about feedback
2. could be a well-intentioned member of the forum or could be a circuit worker (someone that works the SEO forums circuit with the same post or announcement)

Reviewers

1. may find or miss the most noteworthy features (or flaws) of the product
2. may find or miss strengths and weaknesses in a product
3. may or may not be biased in favor or against what he/she is reviewing

As we can see, providing feedback through a forum is one of the most difficult tasks. That’s why I call this “shooting at a moving target”. Without identifying products, services or companies


1. What is your take on the above?
2. What do you expect from new SEO/SEM products or services?
3. What do you like/hate the most from new SEO/SEM products or services?
4. What do you expect from beta test reviewers and submitters?
5. What are the top no-no’s about products or services you have reviewed or are beta testing?

As previously mentioned, the purpose of this thread is to serve as reference point for both submitters and beta reviewers. In this way the thread can be revisited before seeking or providing honest feedback. Most definitely this is not a bashing corner.

Think of it as a “wish list”, “do’s-and-don’ts list” or a “give me your two cents” corner.

Here are my two cents.

Some SEO/SEM specialists and marketers know, talk about or promote the fact that a given marketing strategy cannot be applied to multiple situations. The notion in the sense that “one size fits all” is simply plain wrong. What surprise me is to see the very same marketers adopting this practice when promoting their own products or services. Go figure!

For example, seeking contact information from clients (usernames, passwords, emails, urls, phone numbers, etc) is a standard and fair practice. This works well when a company-client relationship is established or about to be established (e.g., a client about to download software)

However, many users -including myself- choke on the idea that in order to provide beta test feedback or even have a little test drive they have to provide log-in information. You can disagree with me, but I find this practice a huge marketing no-no, especially with companies I never heard about before.

In these days of security flaws, spyware, and spamware it doesn’t hurt to be on a permanent alert. My advice to prospective reviewers, submitters, and users is this: Be an educated consumer. Statement like “Trust us. We don’t use spyware toolbars” insures nothing.

Before using such products or services or downloading anything, use your common sense, as you are on your own risk. Honestly think, would you facilitate information over the phone to someone you never heard about before? Then, why do you want to do this online?

Feel free to add your “wish list”, “do’s-and-don’ts list”, “two cents” or to disagree with me. The more we discuss these issues the more we learn from the experience.


Orion

[Chris Boggs]

I was looking at the FAQ's and found the Beta Test forum info, and therefore I deleted my question about this.

I have another question: What is a "sticky thread?" I have not found the info on that yet...

[mat]

Erm, it's a thread that stays locked above others in such a way that it's always at the top of the page.

[Edited due to being dumber than ...]

It's also a thread that is supposed to grab eyeballs and avoid lots of guideline thwarting/repetitive newbie posts. Good luck with that. Sic.

[Chris Boggs]

thanks Mat for the explanation! makes sense now...

[orion]

FOR REVIEWERS

When reviewing a software or tool


1. be on-topic, stay on-topic. If you want to review a particular software or tool, open a new thread. Trying to review a new tool in a thread that already discusses a product/service not only diverts the current discussion but creates unnecessary noise in the thread.

However, if you want to compare one tool with another in the same thread without trying to review boths, that's fair as long as is relevant to the discussion.


2. be objective, especially when comparing between similar products/services already available.


3. less is more. Be specific. Avoid unnecessary qualifiers.


Orion


PS Feel free to add your two cents or make any suggestion.

[orion]

FOR REVIEWERS

When reviewing, you may want to device an evaluation instrument. Two of the most common instruments are

1. checklists
2. scoring systems

Checklists revisit itemized observables (features, instructions, inputs/outputs, etc). Scoring systems are designed to quantify checklists. Academics have found that merging both into a single instrument saves time and simplifies the reviewing process.

Without pretending to cover all aspects or details of evaluations, here is a crash course on these instruments.

1. Create a scoring scale.
2. Declare in positive mode the observables.
3. Apply the scale to each observable.

That's all! An example follows.

Let say you want to review a software tool and you want to evaluate its usefulness, response time, and price.

First, write your own scoring scale. You decide the scale weights and nature. For example,

Strongly Agree = 4
Agree = 3
Disagree = 2
Strongly Disagree = 1
Not Applicable = NA

In this case the scale spreads from 1 to 4. It is entirely up to you if you want to add more grades to the scale; e.g., "No Comments", "Fails Completely = 0", etc.

Next, observables are declared and weights are assigned. To avoid double negations/rejections, do not declare observables in negative mode. For this example I can write

1. Software is useful = 3
2. Response time is acceptable = 4
3. Price is competitive = 2

Thus, this evaluation means that I agree that the software is useful, I strongly agree that the response time is acceptable, and that I disagree that the price is competitive.

You can go a step further and

a. assign a final (overall) grade based on the total scores.
b. add special comments, explanations, references, and the like.
c. include a T chart.

A T chart is just a two-column table, with one column labeled as PLUS and the other as DELTA. In PLUS, you state positive features -"pluses"- of the software or positive comments. In DELTA, you state features and things that could be changed, enhanced or improved. Some T Charts label this column using other terms such as CONS, FLAWS, etc. I like the "delta" term since is a PC thing (politically correct).

Regardless of how do you want to evaluate or review softwares, tools, sites, etc, soon you will realize that having a system makes things a lot easier and saves time.

What do you think?


Orion

[Chris Boggs]

good stuff Orion. If only I could make all life's decisions based on such study

This is a good way to train the mind for effective critical thinking. The emphasis on positive statements is certainly important, and will help to avoid rejections as you point out.

I'll try it out soon!

[orion]

Hi, Chris

Thank you for such kind words. This approach not only works with software reviews but with anything that needs to be evaluated. Indeed, we can use this type of "positive" evaluations when conducting complimentary site reviews for prospective clients.

I have found that

1. clients are more receptive to this kind of evaluations
2. clients feel more confortable with objectives methodologies
3. are less inclined to overreact to negative findings
4. it gives them a feeling of how their sites compare with competitors' sites
5. helps with GAP analyses

I just hope more SEOs find some value in critical thinking methodologies. That'll make my day, today.

Orion

[orion]

FOR REVIEWERS


REVIEWING SEARCH TOOLS

Reviewing search tools is slightly different from reviewing business intelligence or marketing tools.

When evaluating search tools (search engines, site searches, meta-engines), try to address the following areas

INDEX SIZE; i.e., number of documents returned for popular queries

RELEVANCY; i.e., whether users find what they need.

SPEED; i.e., response time.

USABILITY; i.e., if users understand and navigate with a minimum of instructions or no instructions at all.

FEATURES; i.e. the lack or presence of advanced search features.

LAYOUT; i.e., whether sections are visually defined, proper use of colors; whether add-ons and advanced features are on the way of usability.

NATURAL SEARCHES; i.e, whether the system returns natural language answers to questions (e.g., Where is San Diego? What’s the capital of Mexico? Who is the president of the United States?, etc).

QUERY MODES; i.e., the lack or presence of query modes.

Feel free to add other areas relevant to the technology associated to the tool under consideration. Avoid reviewing business models or revenue models as this is a section dedicated to technology reviews, mostly based on your own user's experience.


Orion

[orion]

FOR SUBMITTERS

The Beta Test Section is dedicated to informally announce new search engine technologies or existing search related products, to get honest feedback from forum members.

First impressions count. The amount and quality of feedback received often depend on that first post announcing your tool or product. You may want to consider the following before seeking feedback:

1. Don’t ask for site or page reviews or for beta testing these. So far, this section is for beta testing/reviewing software/tools. Unless this policy changes, please do not ask for reviews of sites/search engines/directories or papers. Technical/algorithmic/heuristic papers can be discussed in the Search Technology and Relevancy section. Don't repeatedly insist in reviewing your tool or software. If someone is really interested in what you have he/she will stop by and provide you with feedback. This is a high-traffic site. If few or no one reviews your tool/software within the first days chances are they may not be interested.

2. Be specific. State which tool/product you want to be reviewed, briefly describe what exactly it does. Avoid marketing language. When possible, include a specific link to the tool/product to be reviewed. Multiple links are not necessary and will be removed.

3. Don't try networking. Don't ask for sellers, resellers or networking/job opportunities. This is not the right place for this.

4. Avoid posting marketing material. This includes announcements of affiliate programs, promos, discount offers, press releases, and the like. A brief mention about your company is ok, an extensive background or description is unnecessary. You are looking for feedback, not at advertising your company. Avoid marketing lingo and qualifiers such as "The best thing out there", "The next super tool" and the like.

5. Seek feedback for one tool per thread. In this way we all stay on-topic. If you want to seek feedback for other tools, feel free to open a new thread.

6. Don’t try to shape or give the impression of trying to influence feedback; e.g. don’t quote what others say about your tool/product/service or give credits to others involved in the R&D of your tool, etc. You are looking for honest feedback, right?

7. Don’t argue with reviewers that may be giving you comments you may not want to hear. Again, remember that you are looking for honest feedback.

8. Don’t drop contact information (email addresses, signature links, phone numbers, etc) as you can always update your profiles with contact information.

9. Don't offer anything in exchange for feedback. This practice is highly questionable.

10. Stay on-topic when you need to describe/clarify to others (reviewers included) what your tool does. This says it all.

From time to time check this thread for additional updates and informational material.

Orion

[randfish]

Originally Posted by orion[B

7. Don’t argue with reviewers[/b] that may be giving you comments you may not want to hear. Again, remember that you are looking for honest feedback.

The rest of the items seem clear and reasonable, but this one doesn't. No arguing with reviewers is a ludicrous and untenable expectation. The whole idea is to have discussions about the tools. If folks can't argue their point of view, the whole concept of a 'discussion' forum is lost.

If you mean hostile or acrimonious attacks, fine, but then please change this language as right now, it would be against terms of services to point out any errors a reviewer makes.

[orion]

If you mean hostile or acrimonious attacks, fine, but then please change this language as right now, it would be against terms of services to point out any errors a reviewer makes.

Precisely, that is what it means. If you read again carefully I mentioned that it's OK to clarify what exactly your tool does, without engaging in killing the messenger. See Point 10 (Stay on-topic)

Orion

[damon]

Arguing with people giving you advice will stifle opinions and defeat the purpose in asking.

If you disagree with what someone thinks of your site, then ask yourself why they have a different opinion from your own. It's your baby, so it is very easy to ignore its faults.

Some people, mostly on other forums, don't have very insightful or intelligent advice. If ten people reply to your thread and only one of the says it sucks, then you can probably safely ignore that opinion. Rebutting it will, however, discourage others from honestly giving their worthwhile opinions.

Be flattered if someone takes the time to look at your site and spend their time evaluating your site--you're the only one getting anything out of the exchange.

d.