What is Keyword Competitiveness?

[orion]

Many SEOs/SEMs in the industry have defined the competitiveness of a keyword in many different ways. Depending on who you ask, keyword competitiveness means different things for different people. For instance,

1. Some have stated that search results is a measure of keyword competitiveness (KC).

2. Others argue whether or not the search volume from Overture or WordTracker is a fair indicator of keyword competitiveness.

3. Many claim that the search results from Google combined with the keyword volume from Overture provides a better estimate of the competitiveness of a term or phrase(s).

4. Even others use a composite metric from keyword tracking tools coming from dissimilar meta engines. This is equivalent to combining dissimilar analytics and business intelligence metrics from dissimilar media outlets.

In 3 and 4, combining two different or more metrics, some representing document counts and others representing query volume from dissimilar databases (Google with Overture or several meta engines), seem to be an exercise in futility: e.g., two dissimilar analytics from two different stores are combined and taken for a fair metric. Surprisingly, many SEOs/SEMs use and defend this approach, even when the arguments are based on formulas made out of thin air. Purely and simply: based on speculations.

Building a mathematical model that account for document counts, query counts and other variables seem a formidable task.

What do you think? Is there such a mathematical model out there for KC or in the making as you read this? If so, what are the scientific basis of such model?


Orion

[Nacho]

Outstanding post Orion!

Another thing that I've noticed is that people take whatever measurements mentioned in 1 - 4 in your post and determine conclusions to be static over time, which completely untrue. Keywords are similar to products, stock qutoes and many other elements out there that can move over time. A good example can be a movie (eg. "Troy", "Titanic" or whatever), where the keyword volume for "titanic" may show thousands or even millions of searches for a given month but the following drops like fly. Is the keyword still competitive? how can it be determined over time?

[Michael Martinez]

I have relied upon rule-of-thumb determinations for competitiveness because there are some search terms which are highly competitive while showing fewer than 1,000,000 hits in SERPs. Why do I say they are competitive? Because I know the other Web site operators and I know they have highly optimzied their sites. It's virtually impossible for anyone else to break into those search terms because we old dogs have fenced off the top turf.

While I generally agree that anything with more than 1,000,000 raw hits is probably relatively competitive, to be sure I run the query again with quotes around the expression. Sometimes, a huge percentage of hits falls away, and I get a better picture of how many sites are trying to be relevant. I have seen some searches change from 2,000,000+ hits to fewer than 10,000 hits.

I also look at the keyword traffic from WordTracker, Overture, and others. If there are a lot of searches for a phrase, AND a lot of qualified hits on the quoted query, then I feel comfortable saying the term is highly competitive.

But, to me, there is a level of super-competition where nearly everyone involved in a particular industry (or at least all the serious players) is actively optimizing and pursuing aggressive strategies. I'm talking about SEO, adult entertainment, real estate, auto sales, and a few other deeply embattled industries.

A recent interview with two search engine designers (I will look up the reference later and post it in a followup) had one claiming that 80% of all queries are non-commercial in nature. He seemed to be basing that assessment on information about Google's Adwords program. I don't know if that kind of data is generally available, but when questioned about the 80% figure, he said that 80% of all queries run on Google do not generate any Adwords results.

While I have long maintained that most people are NOT (at any given time) searching for commercial products and services, I think this 80% figure (if it holds up to closer scrutiny, assuming such scrutiny is possible) supports what I have been saying for years: that most search terms are NOT highly competitive because the SEO industry hasn't been hitting on them.

[xan]

I would check top ranking sites and devise my own method for doing this. It is a mathematical method like evrything is really. Picking out the most searched for words isn't my biusiness, but I guess you have to watch what queries come into your database and clock it.

[Michael Martinez]

The interview I referred to was Mike Grehan's conversation with Jim Lanzone and Apostolos Gerasoulis.

[I, Brian]

Definitely a good topic area - and as a brief answer I'd say measuring factors via experience rather than via hard numbers.

Number of results for a query isn't necessarily indicative that anyone is searching for those search terms. And just because an area is non-commercial doesn't mean to say that it's easy game for SEO purposes - old and out of date university pages can be real pains to work against.

[DarkMatter]

Among other things, I look at how many sites link to the top ranked sites and whether those sites look "optimized". Granted that good on-page optimization should not be apparent, but when I start seeing keyword loaded title tags and headings all over I assume it's more competitive.

[NTR]

Thank you!

This is an excellent discussion.

DanThies had a post last week on keywords, and I brought up the topic of Keyword competitiveness being skewed if it is based in anyway on what we now consider to be "popularity". (Your points as listed above)

Originally Posted by orion

Building a mathematical model that account for document counts, query counts and other variables seem a formidable task. What do you think? Is there such a mathematical model out there for KC or in the making as you read this? If so, what are the scientific basis of such model?

I'm not even sure that a formula could exist, unless the search engines are open with their numbers - creating a stock market of sorts for keywords - and build analysis in much of the same way. For now, here are my conclusions on the current state of finding "popularity":

To quote my own comments from Dan's post: In the end it will be those in verticals with access to log files across their market that will know what true keyword “popularity” is and/or those in this industry that can figure out what the true margin of error is in “popularity”, who will be the most successful.

[randfish]

Orion, although it wouldn't stand up to scientific scrutiny, the tool I made a while ago to measure KW Difficulty uses the following factors to measure (including data about the top 10 competitors for the term/phrase):

Times Searched Last Month
Results taken from Overture's Keyword Suggestion Tool. This metric helps to determine how much expected traffic a phrase/term will have. Although Overture's tool may be somewhat inaccurate, for relative purposes, it serves well.

# of Results for Search @ Google in "Quotes"
Results taken from a search @ Google for the keyword phrase in quotes. This metric allows us to see how many pages in Google's index have the exact phrase somewhere on the page or in anchor text pointing to them.

# of Results for AllinTitle Search @ Google
Results taken from a search @ Google for allintitle:keyword phrase. This metric helps to determine how many pages in Google's index have all of the target terms in the title of the page.

# of Results for Intitle Inanchor Search @ Google
Results taken from a search @ Google for intitle:term1 intitle:term2 inanchor:term1 inanchor:term2, etc. This metric allows us to see how many pages in Google's index have the term in botht the title of the page and in anchor text pointing to the page. This is an excellent way to measure the number of 'serious' competitors a keyword phrase has.

Top 3 Bids @ Overture
Results are taken from your entries on the previous page, which come from Overture's View Bids Tool. This metric gives us a good feel for how valuable a term is to the market. We measure the top 3 bids, because a very high initial bid with considerably lower 2nd & 3rd bidders is less competitive than a term for which all three bids are high and in close proximity.

Strength of Competitors' Site's Backlinks
Results are taken from a search @ Yahoo! for linkdomain:url.com -site:url.com. This metric allows us to see how many unique, external pages are pointing to the site.

Strength of Competitors' Internal Links to Page
Results are taken from a search @ Yahoo! for link:url.com/page.html site:url.com. This metric tells us how many pages in the site are pointing to the target page. For example, if the page is a tertiary level page inside a site vs. a top level page, it can reveal how powerful the site/page's internal links are.

Strength of Competitors' External Links to Page
Results are taken from a search @ Yahoo! for link:url.com/page.html -siteurl.com. This metric allows us to see how many external links point directly to the ranking page in question. For top-level domains, this result will be the same as #6.

Strength of Competitors' Pages PR
Results are taken from the average of the toolbar PR for the top 10 ranking pages. This metric allows us to see how 'important' the external & internal links are to a given page.

Strength of Competitors' Site's PR
Results are taken from the average of the toolbar PR for the top 10 ranking sites. This metric lets us see the strength & 'importance' of the sites in the top ten.

Strength of Competitors' Size
Results are taken from the average of a search site:url.com @ Google. The metric provides us with information about the relative size of the top 10 sites ranking for the term.

% of TLDs in Top 10 Results
Results are taken by comparing the number of top-level URLs to internal pages in the top 10. The more TLDs, the more difficult it will be to rank well for the keyword phrase in question.

[St0n3y]

As to looking only at the competition of a keyword phrase, we tend to simply look at the sites dominating the top positions. We don't care how "optimized" they are but rather consider the authoritative nature of those sites.

If you're trying to get top ranking for "summer clothing" and Gap, Target, Walmart and the like hold the top 10 positions, you can probably forget about achieving first page placement anytime soon.

Another factor is to consider the relevance of the sites holding the top spots. Again, if searching for "summer clothing" and not-so relevant authoritative sites come up, then you've got a decent shot at it.

Search volume (Wortracker, Overture, etc) is absolutely not an indicator of competition of that keyword, its just a measure of search volume. Many keywords have low search volume but high competitiveness ("exchange hosting" for example).

The number of results returned from Google is a similarly useless indicator (though a fair argument can be made for analyzing that) as a great deal of those sites that come up are not necessarily competing for that phrase or they may not be in the same class of competition (authoritative) level.

[Mel]

The number of results for a search term has not much at all to do with the competitiveness of a search term other than to measure its popularity.

We analyze four factors in determining competitiveness for a quick result:

• Number of results returned for an allintitle: search using the search term in quotes
  
• Number of backlinks in Yahoo + MSN for the top five ranked pages
  
• Top bid prices in Overture for the search term
  
• Gut feeling after reviewing the optimization on the top five ranked pages

For a more detailed analysis a tool like randfishes is good, but IMO the gut feel factor still has to be added.

[orion]

I just came from an activity here in San Diego Overture invited us (Nacho and I). The consensus from the horse mouth and the little horses from Overture is that

a. Keyword competitiveness is unique to each search engine metric. So those that mix and match metrics are way wrong in their approach for keyword competitiveness.

b. keyword competitiveness, as Nacho expressed is time-specific.

c. To my pleasure, they presented a ComScore study in which the data structure

BROADER > NARROWER > SPECIFIC terms

drives searches, with 60% or so of all searches driven by BROADER terms.

This study, coming from third parties and Overture, validates my findings in the On-Topic paper I presented last year.

So, topic specificity must be considered part of any keyword competitive formulae.

Orion

[orion]

Originally Posted by Michael Martinez

IWhile I generally agree that anything with more than 1,000,000 raw hits is probably relatively competitive, to be sure I run the query again with quotes around the expression.

Michael, what you are describing here is the EF-Ratio index.


Orion

[Mel]

While from a theoretical approach the competitiveness of a keyword may be engine specific it is not practical IMO to use such an approach for determining keywords to be used in a campaign which will of course run across all engines worldwide.

Again from the practical approach, it is not IMO necessary for most SEOs to have a handle on how competitive a keyword is down to five decimal points. It is usually good enough to know that this word is more competitive than that word and then do a comparasion of the benfits related to each.

[orion]

Originally Posted by Mel

While from a theoretical approach the competitiveness of a keyword may be engine specific it is not practical IMO to use such an approach for determining keywords to be used in a campaign which will of course run across all engines worldwide.

Again from the practical approach, it is not IMO necessary for most SEOs to have a handle on how competitive a keyword is down to five decimal points. It is usually good enough to know that this word is more competitive than that word and then do a comparasion of the benfits related to each.

Nah!

1. the results presented are not theoretical but experimental (from ComScore and Overture)
2. the results come from Overture marketing folks presenting at the above activity; i.e. the horse's mouth.

Their own metric validates our research data concerning what is/is not keyword competitiveness.

Mixing metrics from dissimilar search engines databases to come up with a measure of keyword competitiveness is not only wrong, it can also mask variable effects and effect interactions between variables.

More on this I can discuss with anyone that want to talk to me at NY SES.

Orion

[Michael Martinez]

Originally Posted by orion

Michael, what you are describing here is the EF-Ratio index.

Orion

You mean this EF Ratio.

Proposed Definition: Given a query Q=k1 k2 k3...kn consisting of n terms and where each k is a single term. The probability that a search for Q in FIND ALL mode will return documents with the EXACT sequence Q=k1 k2 k3...kn is its EF ratio.

However, I am not interested in the probability represented by this ratio. I don't use the ratio as a predictor of anything.

I am only interested in determining how competitive a given phrase is. If a search with fewer than 1,000,000 results in find all mode reduces to fewer than 10% or less in exact mode, I infer that the expression may not have been highly optimized for across a significant number of pages. It may have a close relative, though, that has spurred intense competition.

Take the expression "real estate" for example. First, the search results without quoting the expression produces around 77,000,000 hits:

http://www.google.com/search?hl=en&q=real+estate

But adding quotes doesn't refine the hit count much:

http://www.google.com/search?num=20&...real+estate%22

I get over 71,000,000 hits. That is a LOT of pages which place the two words close together (I understand that I am not searching for the exact phrase "real estate"). It would be difficult for anyone new to break into and dominate this phrase.

As a random comparison, I'll use the expression "sharper swords". First hits (without quotes) produces fewer than 100,000 results:

http://www.google.com/search?num=20&...sharper+swords

A search using the result produces fewer than 1,000 hits:

http://www.google.com/search?num=20&...rper+swords%22

The top results pages are very similar for both searches, but none of them are really emphasizing "sharper swords". This tells me that no one is optimizing for the phrase "sharper swords" or anything close to it. I am reasonably confident I could create a page targeting on that term and get it to number 1 in the SERPs without having to go to any special effort.

Another example (plucked at random from my chaotic thoughts) is the expression "bagel king":

http://www.google.com/search?num=20&...1&q=bagel+king

Without quotes, I see right away that someone has actually positioned a page for this expression simply because it comes up first in the results. But the results show 211,000 hits.

Applying the quotes, I get fewer than 2,000 hits:

http://www.google.com/search?num=20&...2bagel+king%22

Our friend who was number 1 in the first search is not number 1 in this search. Before looking at his page, I infer that he has not optimized the page as tightly as I would, if I wanted to grab the number 1 slot. Looking at the source code, I feel I could give the guy a run for his money, maybe even take the top slot away from him, without much effort.

Nothing else in the results seems to be optimized, so it appears that one person went after a relatively unstaked phrase.

Another example, conceived of randomly, which I would think has generated some optimization:

http://www.google.com/search?num=20&...q=auto+auction

I get over 8,000,000 hits. By my rule-of-thumb, this is a competitive phrase. However, if you use quotes, you get:

http://www.google.com/search?num=20&...uto+auction%22

403,000 hits. What I infer from this result is that the term is moderately optimized for. A great deal of optimization should have produced more than 800,000 hits in exact mode. But many of the results we see do include "auto auction" in their titles, so there is clearly SOME optimization going on. Based on what I see in the titles, I believe it is possible some of this optimization is directed at slightly different but related phrases.

If I had found fewer than 100,000 hits on the exact mode search, I would have inferred that the phrase is only lightly optimized for, and that perhaps there isn't much in the way of closely related phrases that are optimized for.

But the competition could still be fierce for an unpopular phrase.

Here is an example where the huge number of hits fall away in exact mode. In find all mode, we get almost 22,000,000 hits for "simple plans":

http://www.google.com/search?num=20&...q=simple+plans

But how many people are REALLY trying to optimize for this expression?

http://www.google.com/search?num=20&...imple+plans%22

I get fewer than 50,000 results.

The results, however, are dominated by obviously optimized pages. It would not necessarily be that easy to break into this expression. The EF Ratio is not a predictor of such difficulty.

The purpose of refining a search is to strip away the junk to search for whatever may be fully optimized. The exact mode search uncovers content which may be optimized for either the targeted phrase or something closely related and very similar.

[Mel]

I am saying that your theoretical approach may be fine in the lab, but doesn't cut the mustard in everyday use

[Nacho]

Originally Posted by Michael Martinez

The EF Ratio is not a predictor of such difficulty.

I disagree with your statement Michael. I believe the EF Ratio may be one among other elements combined or not with the ones mentioed in 1-4 in post #1 that may give you an indicator (call it "feedback" if you wish) of keyword competitiveness. However, from tests we have done, the EF Ratio does not provide enough evidence to be a stand alone metric of KC, but YES it is a powerful one.

For example, I'm sure we could come up with an example from the "viagra" community of keywords (as KW1 combined with some other KW2) with an X EF Ratio that matches the same EF Ratio from a much more less spammy community of keywords derived from websites. Therefore, in that case it wouldn't be such a good indicator, right? But in many other examples it may be cristal clear about it.

BTW, Dr. E. Garcia (Orion) will not be posting this weekend as he is now getting ready to travel to NYC. I strongly recommend going to the Search Algorithm Research & Developments session in SES NYC '05 where he will give and explain more on the subject.

Aside from EF Ratios, which I find facinating, let's come up with other ideas for this thread that might help determine keyword competitiveness. I would not like this thread to only talk about EF Ratios please.

[Michael Martinez]

Originally Posted by Nacho

I disagree with your statement Michael. I believe the EF Ratio may be one among other elements combined or not with the ones mentioed in 1-4 in post #1 that may give you an indicator (call it "feedback" if you wish) of keyword competitiveness.

My statement was not concerned with the competitiveness of the keywords, but rather with the competitiveness of the optimization for specifically targeted niche phrases (the last example I provided shows one such targeted niche).

The low EF Ratio would be a misleading indicator of what is going on with those pages.

My point is that SOMETIMES a small number of people apply the best SEO techniques for a relatively unpopular term.

[Nacho]

Thank you for the clarification Michael.