What is Keyword Competitiveness?

[orion]

Many SEOs/SEMs in the industry have defined the competitiveness of a keyword in many different ways. Depending on who you ask, keyword competitiveness means different things for different people. For instance,

1. Some have stated that search results is a measure of keyword competitiveness (KC).

2. Others argue whether or not the search volume from Overture or WordTracker is a fair indicator of keyword competitiveness.

3. Many claim that the search results from Google combined with the keyword volume from Overture provides a better estimate of the competitiveness of a term or phrase(s).

4. Even others use a composite metric from keyword tracking tools coming from dissimilar meta engines. This is equivalent to combining dissimilar analytics and business intelligence metrics from dissimilar media outlets.

In 3 and 4, combining two different or more metrics, some representing document counts and others representing query volume from dissimilar databases (Google with Overture or several meta engines), seem to be an exercise in futility: e.g., two dissimilar analytics from two different stores are combined and taken for a fair metric. Surprisingly, many SEOs/SEMs use and defend this approach, even when the arguments are based on formulas made out of thin air. Purely and simply: based on speculations.

Building a mathematical model that account for document counts, query counts and other variables seem a formidable task.

What do you think? Is there such a mathematical model out there for KC or in the making as you read this? If so, what are the scientific basis of such model?


Orion

[Nacho]

Outstanding post Orion!

Another thing that I've noticed is that people take whatever measurements mentioned in 1 - 4 in your post and determine conclusions to be static over time, which completely untrue. Keywords are similar to products, stock qutoes and many other elements out there that can move over time. A good example can be a movie (eg. "Troy", "Titanic" or whatever), where the keyword volume for "titanic" may show thousands or even millions of searches for a given month but the following drops like fly. Is the keyword still competitive? how can it be determined over time?

[Michael Martinez]

I have relied upon rule-of-thumb determinations for competitiveness because there are some search terms which are highly competitive while showing fewer than 1,000,000 hits in SERPs. Why do I say they are competitive? Because I know the other Web site operators and I know they have highly optimzied their sites. It's virtually impossible for anyone else to break into those search terms because we old dogs have fenced off the top turf.

While I generally agree that anything with more than 1,000,000 raw hits is probably relatively competitive, to be sure I run the query again with quotes around the expression. Sometimes, a huge percentage of hits falls away, and I get a better picture of how many sites are trying to be relevant. I have seen some searches change from 2,000,000+ hits to fewer than 10,000 hits.

I also look at the keyword traffic from WordTracker, Overture, and others. If there are a lot of searches for a phrase, AND a lot of qualified hits on the quoted query, then I feel comfortable saying the term is highly competitive.

But, to me, there is a level of super-competition where nearly everyone involved in a particular industry (or at least all the serious players) is actively optimizing and pursuing aggressive strategies. I'm talking about SEO, adult entertainment, real estate, auto sales, and a few other deeply embattled industries.

A recent interview with two search engine designers (I will look up the reference later and post it in a followup) had one claiming that 80% of all queries are non-commercial in nature. He seemed to be basing that assessment on information about Google's Adwords program. I don't know if that kind of data is generally available, but when questioned about the 80% figure, he said that 80% of all queries run on Google do not generate any Adwords results.

While I have long maintained that most people are NOT (at any given time) searching for commercial products and services, I think this 80% figure (if it holds up to closer scrutiny, assuming such scrutiny is possible) supports what I have been saying for years: that most search terms are NOT highly competitive because the SEO industry hasn't been hitting on them.

[xan]

I would check top ranking sites and devise my own method for doing this. It is a mathematical method like evrything is really. Picking out the most searched for words isn't my biusiness, but I guess you have to watch what queries come into your database and clock it.

[Michael Martinez]

The interview I referred to was Mike Grehan's conversation with Jim Lanzone and Apostolos Gerasoulis (http://www.e-marketing-news.co.uk/Feb05/apostolos.html).

[I, Brian]

Definitely a good topic area - and as a brief answer I'd say measuring factors via experience rather than via hard numbers.

Number of results for a query isn't necessarily indicative that anyone is searching for those search terms. And just because an area is non-commercial doesn't mean to say that it's easy game for SEO purposes - old and out of date university pages can be real pains to work against.

[DarkMatter]

Among other things, I look at how many sites link to the top ranked sites and whether those sites look "optimized". Granted that good on-page optimization should not be apparent, but when I start seeing keyword loaded title tags and headings all over I assume it's more competitive.

[NTR]

Thank you!

This is an excellent discussion.

DanThies had a post (http://www.sitepoint.com/blog-post-view.php?id=235704#comments) last week on keywords, and I brought up the topic of Keyword competitiveness being skewed if it is based in anyway on what we now consider to be "popularity". (Your points as listed above)

Building a mathematical model that account for document counts, query counts and other variables seem a formidable task. What do you think? Is there such a mathematical model out there for KC or in the making as you read this? If so, what are the scientific basis of such model?

I'm not even sure that a formula could exist, unless the search engines are open with their numbers - creating a stock market of sorts for keywords - and build analysis in much of the same way. For now, here are my conclusions on the current state of finding "popularity":

To quote my own comments from Dan's post: In the end it will be those in verticals with access to log files across their market that will know what true keyword “popularity” is and/or those in this industry that can figure out what the true margin of error is in “popularity”, who will be the most successful.

[randfish]

Orion, although it wouldn't stand up to scientific scrutiny, the tool (http://www.socengine.com/seo/tools/keyword-difficulty-tool.php) I made a while ago to measure KW Difficulty uses the following factors to measure (including data about the top 10 competitors for the term/phrase):

Times Searched Last Month
Results taken from Overture's Keyword Suggestion Tool. This metric helps to determine how much expected traffic a phrase/term will have. Although Overture's tool may be somewhat inaccurate, for relative purposes, it serves well.

# of Results for Search @ Google in "Quotes"
Results taken from a search @ Google for the keyword phrase in quotes. This metric allows us to see how many pages in Google's index have the exact phrase somewhere on the page or in anchor text pointing to them.

# of Results for AllinTitle Search @ Google
Results taken from a search @ Google for allintitle:keyword phrase. This metric helps to determine how many pages in Google's index have all of the target terms in the title of the page.

# of Results for Intitle Inanchor Search @ Google
Results taken from a search @ Google for intitle:term1 intitle:term2 inanchor:term1 inanchor:term2, etc. This metric allows us to see how many pages in Google's index have the term in botht the title of the page and in anchor text pointing to the page. This is an excellent way to measure the number of 'serious' competitors a keyword phrase has.

Top 3 Bids @ Overture
Results are taken from your entries on the previous page, which come from Overture's View Bids Tool. This metric gives us a good feel for how valuable a term is to the market. We measure the top 3 bids, because a very high initial bid with considerably lower 2nd & 3rd bidders is less competitive than a term for which all three bids are high and in close proximity.

Strength of Competitors' Site's Backlinks
Results are taken from a search @ Yahoo! for linkdomain:url.com -site:url.com. This metric allows us to see how many unique, external pages are pointing to the site.

Strength of Competitors' Internal Links to Page
Results are taken from a search @ Yahoo! for link:url.com/page.html site:url.com. This metric tells us how many pages in the site are pointing to the target page. For example, if the page is a tertiary level page inside a site vs. a top level page, it can reveal how powerful the site/page's internal links are.

Strength of Competitors' External Links to Page
Results are taken from a search @ Yahoo! for link:url.com/page.html -siteurl.com. This metric allows us to see how many external links point directly to the ranking page in question. For top-level domains, this result will be the same as #6.

Strength of Competitors' Pages PR
Results are taken from the average of the toolbar PR for the top 10 ranking pages. This metric allows us to see how 'important' the external & internal links are to a given page.

Strength of Competitors' Site's PR
Results are taken from the average of the toolbar PR for the top 10 ranking sites. This metric lets us see the strength & 'importance' of the sites in the top ten.

Strength of Competitors' Size
Results are taken from the average of a search site:url.com @ Google. The metric provides us with information about the relative size of the top 10 sites ranking for the term.

% of TLDs in Top 10 Results
Results are taken by comparing the number of top-level URLs to internal pages in the top 10. The more TLDs, the more difficult it will be to rank well for the keyword phrase in question.

[St0n3y]

As to looking only at the competition of a keyword phrase, we tend to simply look at the sites dominating the top positions. We don't care how "optimized" they are but rather consider the authoritative nature of those sites.

If you're trying to get top ranking for "summer clothing" and Gap, Target, Walmart and the like hold the top 10 positions, you can probably forget about achieving first page placement anytime soon.

Another factor is to consider the relevance of the sites holding the top spots. Again, if searching for "summer clothing" and not-so relevant authoritative sites come up, then you've got a decent shot at it.

Search volume (Wortracker, Overture, etc) is absolutely not an indicator of competition of that keyword, its just a measure of search volume. Many keywords have low search volume but high competitiveness ("exchange hosting" for example).

The number of results returned from Google is a similarly useless indicator (though a fair argument can be made for analyzing that) as a great deal of those sites that come up are not necessarily competing for that phrase or they may not be in the same class of competition (authoritative) level.

[Mel]

The number of results for a search term has not much at all to do with the competitiveness of a search term other than to measure its popularity.

We analyze four factors in determining competitiveness for a quick result:

Number of results returned for an allintitle: search using the search term in quotes

Number of backlinks in Yahoo + MSN for the top five ranked pages

Top bid prices in Overture for the search term

Gut feeling after reviewing the optimization on the top five ranked pages


For a more detailed analysis a tool like randfishes is good, but IMO the gut feel factor still has to be added.

[orion]

I just came from an activity here in San Diego Overture invited us (Nacho and I). The consensus from the horse mouth and the little horses from Overture is that

a. Keyword competitiveness is unique to each search engine metric. So those that mix and match metrics are way wrong in their approach for keyword competitiveness.

b. keyword competitiveness, as Nacho expressed is time-specific.

c. To my pleasure, they presented a ComScore study in which the data structure

BROADER > NARROWER > SPECIFIC terms

drives searches, with 60% or so of all searches driven by BROADER terms.

This study, coming from third parties and Overture, validates my findings in the On-Topic paper I presented last year.

So, topic specificity must be considered part of any keyword competitive formulae.

Orion

[orion]

IWhile I generally agree that anything with more than 1,000,000 raw hits is probably relatively competitive, to be sure I run the query again with quotes around the expression.
Michael, what you are describing here is the EF-Ratio index.


Orion

[Mel]

While from a theoretical approach the competitiveness of a keyword may be engine specific it is not practical IMO to use such an approach for determining keywords to be used in a campaign which will of course run across all engines worldwide.

Again from the practical approach, it is not IMO necessary for most SEOs to have a handle on how competitive a keyword is down to five decimal points. It is usually good enough to know that this word is more competitive than that word and then do a comparasion of the benfits related to each.

[orion]

While from a theoretical approach the competitiveness of a keyword may be engine specific it is not practical IMO to use such an approach for determining keywords to be used in a campaign which will of course run across all engines worldwide.

Again from the practical approach, it is not IMO necessary for most SEOs to have a handle on how competitive a keyword is down to five decimal points. It is usually good enough to know that this word is more competitive than that word and then do a comparasion of the benfits related to each.Nah!

1. the results presented are not theoretical but experimental (from ComScore and Overture)
2. the results come from Overture marketing folks presenting at the above activity; i.e. the horse's mouth.

Their own metric validates our research data concerning what is/is not keyword competitiveness.

Mixing metrics from dissimilar search engines databases to come up with a measure of keyword competitiveness is not only wrong, it can also mask variable effects and effect interactions between variables.

More on this I can discuss with anyone that want to talk to me at NY SES.

Orion

[Michael Martinez]

Michael, what you are describing here is the EF-Ratio index.

Orion

You mean this EF Ratio (http://www.miislita.com/semantics/c-index-4.html).

Proposed Definition: Given a query Q=k1 k2 k3...kn consisting of n terms and where each k is a single term. The probability that a search for Q in FIND ALL mode will return documents with the EXACT sequence Q=k1 k2 k3...kn is its EF ratio.

However, I am not interested in the probability represented by this ratio. I don't use the ratio as a predictor of anything.

I am only interested in determining how competitive a given phrase is. If a search with fewer than 1,000,000 results in find all mode reduces to fewer than 10% or less in exact mode, I infer that the expression may not have been highly optimized for across a significant number of pages. It may have a close relative, though, that has spurred intense competition.

Take the expression "real estate" for example. First, the search results without quoting the expression produces around 77,000,000 hits:

http://www.google.com/search?hl=en&q=real+estate

But adding quotes doesn't refine the hit count much:

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=%2B%22real+estate%22

I get over 71,000,000 hits. That is a LOT of pages which place the two words close together (I understand that I am not searching for the exact phrase "real estate"). It would be difficult for anyone new to break into and dominate this phrase.

As a random comparison, I'll use the expression "sharper swords". First hits (without quotes) produces fewer than 100,000 results:

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=sharper+swords

A search using the result produces fewer than 1,000 hits:

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=%2B%22sharper+swords%22

The top results pages are very similar for both searches, but none of them are really emphasizing "sharper swords". This tells me that no one is optimizing for the phrase "sharper swords" or anything close to it. I am reasonably confident I could create a page targeting on that term and get it to number 1 in the SERPs without having to go to any special effort.

Another example (plucked at random from my chaotic thoughts) is the expression "bagel king":

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=bagel+king

Without quotes, I see right away that someone has actually positioned a page for this expression simply because it comes up first in the results. But the results show 211,000 hits.

Applying the quotes, I get fewer than 2,000 hits:

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=%2B%22bagel+king%22

Our friend who was number 1 in the first search is not number 1 in this search. Before looking at his page, I infer that he has not optimized the page as tightly as I would, if I wanted to grab the number 1 slot. Looking at the source code, I feel I could give the guy a run for his money, maybe even take the top slot away from him, without much effort.

Nothing else in the results seems to be optimized, so it appears that one person went after a relatively unstaked phrase.

Another example, conceived of randomly, which I would think has generated some optimization:

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=auto+auction

I get over 8,000,000 hits. By my rule-of-thumb, this is a competitive phrase. However, if you use quotes, you get:

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=%2B%22auto+auction%22

403,000 hits. What I infer from this result is that the term is moderately optimized for. A great deal of optimization should have produced more than 800,000 hits in exact mode. But many of the results we see do include "auto auction" in their titles, so there is clearly SOME optimization going on. Based on what I see in the titles, I believe it is possible some of this optimization is directed at slightly different but related phrases.

If I had found fewer than 100,000 hits on the exact mode search, I would have inferred that the phrase is only lightly optimized for, and that perhaps there isn't much in the way of closely related phrases that are optimized for.

But the competition could still be fierce for an unpopular phrase.

Here is an example where the huge number of hits fall away in exact mode. In find all mode, we get almost 22,000,000 hits for "simple plans":

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=simple+plans

But how many people are REALLY trying to optimize for this expression?

http://www.google.com/search?num=20&hl=en&lr=&newwindow=1&safe=off&c2coff=1&q=%2B%22simple+plans%22

I get fewer than 50,000 results.

The results, however, are dominated by obviously optimized pages. It would not necessarily be that easy to break into this expression. The EF Ratio is not a predictor of such difficulty.

The purpose of refining a search is to strip away the junk to search for whatever may be fully optimized. The exact mode search uncovers content which may be optimized for either the targeted phrase or something closely related and very similar.

[Mel]

I am saying that your theoretical approach may be fine in the lab, but doesn't cut the mustard in everyday use

[Nacho]

The EF Ratio is not a predictor of such difficulty.
I disagree with your statement Michael. I believe the EF Ratio may be one among other elements combined or not with the ones mentioed in 1-4 in post #1 that may give you an indicator (call it "feedback" if you wish) of keyword competitiveness. However, from tests we have done, the EF Ratio does not provide enough evidence to be a stand alone metric of KC, but YES it is a powerful one.

For example, I'm sure we could come up with an example from the "viagra" community of keywords (as KW1 combined with some other KW2) with an X EF Ratio that matches the same EF Ratio from a much more less spammy community of keywords derived from websites. Therefore, in that case it wouldn't be such a good indicator, right? But in many other examples it may be cristal clear about it.

BTW, Dr. E. Garcia (Orion) will not be posting this weekend as he is now getting ready to travel to NYC. I strongly recommend going to the Search Algorithm Research & Developments (http://www.jupiterevents.com/sew/winter05/agenda.html#345-2) session in SES NYC '05 where he will give and explain more on the subject.

Aside from EF Ratios, which I find facinating, let's come up with other ideas for this thread that might help determine keyword competitiveness. I would not like this thread to only talk about EF Ratios please.

[Michael Martinez]

I disagree with your statement Michael. I believe the EF Ratio may be one among other elements combined or not with the ones mentioed in 1-4 in post #1 that may give you an indicator (call it "feedback" if you wish) of keyword competitiveness.

My statement was not concerned with the competitiveness of the keywords, but rather with the competitiveness of the optimization for specifically targeted niche phrases (the last example I provided shows one such targeted niche).

The low EF Ratio would be a misleading indicator of what is going on with those pages.

My point is that SOMETIMES a small number of people apply the best SEO techniques for a relatively unpopular term.

[Nacho]

Thank you for the clarification Michael. :)

[xan]

The whole thing is a guess. None of it can be accurately measured unless you are the one who owns the search engine and you keep a log of every query made. This will also only show what happens via your SE not others. The overall picture is always an estimation. If you own 60% of the search market, then you are most likely to give a closer measure than the others.

I agree with Orion on this.

[DanThies]

I wish I had more time today, because I have a lot more to say on this topic... since it is my business. :D We include several competition metrics in our keyword reports, but we're limited by what's practical to extract and compute.

The E/F ratio is interesting when you drill down into longer search terms, but more so when you look at it within a smaller population of documents. For example, if you look at the entire Google database to compare "viagra" with the longer and related terms that float around it, you get a very different result than if you compare the E/F ratio within the pages that constitute the 1000-page result set for "viagra." Using a subset of the web to calculate E/F ratio puts it into sharper focus, but of course requires substantial computing resources. This is related to the use of c-indexes, but not identical.

What I see a lot of marketers doing, though, is overanalyzing the competition when developing keyword/content strategies. Identifying that set of search terms which best reaches the target audience *is* the proper starting point for a search engine marketing campaign. Some terms will be worth the effort to target with SEO, some will be cost-effective for PPC campaigns, and the rest will still need to be part of the site's content in order to satisfy visitors.

For most websites, a more productive area would be to improve the site's content and conversion rates, thereby raising the level of acceptable effort for SEO and the maximum profitable bid for PPC campaigns. I rarely consult on an SEM campaign where we don't find that much of the effort must take place in making more happen with the existing and resulting visitor traffic.

It's easy to narrow a PPC campaign (targeting better) to artificially improve the site's conversion rate, much harder to actually do the work that's required to make the site financially competitive. But competitiveness has more than one aspect - when you optimize for business results instead of just rankings, a lot of the decisions you make will be very different.

[earlpearl]

From an operating perspective, both Randfish's tool and Mel's methods are terrific tools to use for various keyword phrases. They assist in dissecting the character of sites at the top rankings for various keyword phrases. Critiques of Randfish's efforts at this forum helped to improve the analyses in the current tool creating a better measure.

In fact the tools allow an SEO to more critically look at the characteristics of sites at the top enabling one to determine if it would be possible to attain high rankings and how that might be possible in light of the competition.

Theoretically a search phrase such as "summer clothes" could include both several major retailers and other types of sites with different characteristics that define the site relative to the keyword phrase and upon using these tools one could find that the phrase is one where an SEO practitioner could compete with the large retailers albeit from a different perspective.

As the algo's are further refined and different elements gain more importance as others decrease in importance tools such as these may need to be refined and changed but they do provide a critical look into keyword competitiveness.

Dave

[Mel]

The whole thing is a guess. None of it can be accurately measured unless you are the one who owns the search engine and you keep a log of every query made. This will also only show what happens via your SE not others. The overall picture is always an estimation. If you own 60% of the search market, then you are most likely to give a closer measure than the others.

I agree with Orion on this.

While there are certainly advantages of having the information that a search engine has available to you I seriously doubt if any search engine has ever looked at the competitiveness of keywords in the same way that those who have to achive rankings for those keywords do.

While the number of searches and the exact details of what it takes to rank for those searches are available to the search engines, I really doubt that they are at all concerned with who ranks for what so long as the results are relevant.

In much the same way, the theory behind the analysis of keyword competitiveness is doubtless useful in gaining a better understanding of it, if you give me the choice of two prospective new SEO hires, one with a Phd in Search Technology and one who has a track record for being able to judge intuitively which are the best words to optimize for, I'll take the guy with the good gut feel every time.

[xan]

While there are certainly advantages of having the information that a search engine has available to you I seriously doubt if any search engine has ever looked at the competitiveness of keywords in the same way that those who have to achive rankings for those keywords do.

While the number of searches and the exact details of what it takes to rank for those searches are available to the search engines, I really doubt that they are at all concerned with who ranks for what so long as the results are relevant.

In much the same way, the theory behind the analysis of keyword competitiveness is doubtless useful in gaining a better understanding of it, if you give me the choice of two prospective new SEO hires, one with a Phd in Search Technology and one who has a track record for being able to judge intuitively which are the best words to optimize for, I'll take the guy with the good gut feel every time.


Of course they don't bother Mel. But that is the way to have the most accurate estimate.

[mcanerin]

I think part of the confusion about developing a KCI (Keyword Competitivness Index) is that Keywords are not competitive - Websites are.

It doesn't matter how many websites show up as potential matches to a query - what matters is how many of them are actually optimized for that keyword, and more importantly, how well optimized.

If your goal is to be in the top ten, then you have ten competitors - that's it. It doesn't matter if there are 5 or 5 million runners-up. Yes, the chances are that a KW that has a lot of results is more likely to have better competitors at the top - but not necessarily. There are terms that have lots of results yet no one has optimized for them, and there will be extremely competitive areas with only a few competitors, but extremely well optimized.

I think the best way to measure the competitiveness of a keyword is to measure the competitiveness of the sites that are in the place you want to show up in (ie top 10) for each individual search term, on each different search engine.

All that said, I also think that there is a place for an easy to use metric that quickly gives an SEO a good idea of how much they are going to have to charge the client. ;) It may not be perfect, but a close approximation that's easy to use is often better from a business standpoint than an exact measurment that's extremely difficult and time consuming.

Ian

[St0n3y]

Keywords are not competitive - Websites are.

Completely agree, but I think its important to realize that a website does not have to be "optimized" to be competitive. If the site you're competing against is a known authority in an industry, that site doesn't necessarily have to be optimized for the terms they are ranking for, they just have to be relevant and the authority status carries them. I've seen this quite a bit with government sites getting top spots for certain keywords.

The two main things to consider is how well are the competing web sites optimized, and are they an authority? That'll tell you what you're up against.

[rivest]

If you have spent much time earning a living on the Internet, few things will surprise you. However, I am always amazed with how many companies and individuals make astounding claims without backing them up.

Web hosts and ISPs make such claims all the time: 99% uptime. Some web hosts post on their site live results of their uptime from a third-party website monitoring company. However, those hosts are very few.

Here are four reasons why every web host and ISP (Internet service provider) should post uptime and connectivity statistics from an independent, third-party website monitoring firm.

Monitoring reports are great marketing tools

This is the most obvious reason a web host or ISP should post monitoring reports on their site. Don’t tell me you have 99% uptime. Even the worst hosting companies and ISPs can make that claim, but it does not make it true. Prove it to me. Posting a report right on your home page, or posting a link to the report up front on the home page tells me you not only can make claims, but you can deliver!

Make sure to post your uptime statistics for the last day, week, month and year, so the customers know you are in it for the tong haul.

Web site monitoring reduces technical support work-load

A recent review by Dotcom-Monitor reveals that accessibility calls to a web host’s technical support can be reduced by 30% just by posting the current status of the web site and server as reported by an independent monitoring service.

Often a user will call technical support if he encounters a slow Internet connection or download times, dial-up problems, website accessibility or various computer issues. Users assume the problem is with their web host or their ISP, and call technical support to fix the problem, even if the problem is on their computer. Real time independent connectivity reports eliminate the need for such calls in many cases, by showing what the status of the site is. A sample report at http://www.dotcom-monitor.com/reporting-card.asp gives an idea of how this works.

Website monitoring is excellent customer relations

If it costs up to ten times more to attract a new customer than to keep an existing one, independent website monitoring can help you keep those precious customers.

Nothing is more frustrating for the end-user than to have her site down and nothing is more embarrassing for an ISP to receive a call from customer about a downtime, especially if it is related to a connectivity issue the hosting company or ISP is not even aware of. External monitoring alerts an ISP or web host immediately to problems that often cannot be detected by internal monitoring, allowing the company to address the issue immediately…and hopefully before a customer even notices that a problem exists.

Monitoring cuts the burden of dispute resolution

Corporate hosting clients and Internet access customers often expect certain levels of service and enter into an SLA (service level agreement). Often, SLA contracts impose fines on ISPs and web hosts who do not meet the targets. Determining how those levels are measured can be difficult, but an independent, third-party monitoring service can easily resolve disputes before they begin.

Dotcom-Monitor is among those services that provides an added SLA reporting function, which can be seen at http://www.dotcom-monitor.com/service-level-agreement.asp . Both the host or ISP and the client can view the reports, and reporting intervals can be set based on the terms of the SLA contract.

While every website can benefit from some form of monitoring, no class of business needs more the credibility of an independent monitoring service as much as those who provide access to the Internet for their clients: ISPs and web hosting companies.

[Michael Martinez]

I think keywords can indeed be competitive. No two people think exactly alike. I often find many variations on the same basic phrase when doing keyword research. The more popular variations tend to get most of the optimized Web sites targeting them. I often target less popular variations in the hope of capturing more traffic from otherwise frustrated searches.

Which is not to say that I target ONLY less popular variations. I am sure if I don't qualify that, someone will point out "It's better to go after the popular terms".

[St0n3y]

I think keywords can indeed be competitive.

What exactly are the keywords competing for? Websites are competitive as they are competing for top placement for specific keywords, and I know we all talk in the lingo of competiveness of keywords, but realistically, what are the keywords competing for? Are we out there trying to get people to search more for specific keywords?

[Michael Martinez]

What exactly are the keywords competing for?

You have to shift your frame of reference to understand what I am saying.

Let's take two search terms that have fallen out of search popularity:

"Lord of the Rings movie news"
"LoTR movie news"

Both terms would take you to the same group of Web sites, but mine ranked more consistently for both.

When I ran analyses of these search terms, I found that more people were looking for "lord of the rings movie news" than were looking for "lotr movie news". That actually surprised me, because there was a time when nearly all online Tolkien fans used "LoTR" instead of spelling out "The Lord of the Rings" (another popular variant was TLoTR).

What this told me was that a whole new generation of fans, less informed about the books than the older generation, was searching for information on the movies. They didn't know about the LOTR acronym, so they didn't use it.

Hence, I was able to attract traffic from both groups of searchers (which undoubtedly shared some overlap).

The unique but related phrases represent different approaches that searchers use to find the same information. One phrase will be more popular than the others (so far, that is the way it has turned out in all my analyses). The most popular phrase is usually the one that the optimizers chase. But the top five to ten next most popular phrases MAY generate enough results to make it worthwhile to optimize for them.

If no one is else optimizing for them, you can usually shoot straight to the top and get a LOT of traffic.

So, the keywords are in essence competing for optimization, or for relevant search results.

[St0n3y]

I understand what you're getting at, which is pretty much how the SEO industry views keyword competiveness, I'm just saying that IMO the word competition is more active than passive.

The most popular phrase is usually the one that the optimizers chase. But the top five to ten next most popular phrases MAY generate enough results to make it worthwhile to optimize for them.

If no one is else optimizing for them, you can usually shoot straight to the top and get a LOT of traffic.

If no one esle is optimizing for them then the really are not that competitive, even if they get large amounts of search volum.

[Michael Martinez]

I understand what you're getting at, which is pretty much how the SEO industry views keyword competiveness, I'm just saying that IMO the word competition is more active than passive.

If no one esle is optimizing for them then the really are not that competitive, even if they get large amounts of search volum.

Then you really are not looking at it in the same way I am. The competition is not between the Web sites for visitors, it is between the keywords for attention (from both visitors AND Web sites).

Whether anyone else had optimized for the sites or not really doesn't matter. If the searches both return relevant results, my experience has shown me that most optimizers will just focus on the main (most popular) search.

[Mel]

... But competitiveness has more than one aspect - when you optimize for business results instead of just rankings, a lot of the decisions you make will be very different.

That's a great statement Dan, and one that this discussion seems to be veering away from. Are we discussing Keyword competitiveness from a thoeretical standpoint or from the standpoint of those who have to put the theory into practice and get results for their clients?

[DanThies]

Are we discussing Keyword competitiveness from a thoeretical standpoint or from the standpoint of those who have to put the theory into practice and get results for their clients?
Thanks, Mel!

I don't see how you can define "competitiveness" in any other way than the level of effort or investment required for a given website to compete. There should also be a consideration of risk in any economic model... with 200 SEOs competing to crowd the same narrow channel, you have more downside risk than if it's Mel the lone SEO vs. the huddled masses of webmasters.

To Ian's point, the website will be more or less competitive for a given search term depending on the resources available. Or to look at it another way, depending on what they can do with the resulting traffic. Some sites will have natural competitive advantages (reputation, linking relationships, content syndication, etc.) that make it easier to rank for some terms, and further reduces the necessary level of effort.

[Mel]

Interesting point Dan.

Yes I totally agree that when SEOs discuss keyword competitiveness it has to be from the standpoint of the time and expense that it will take to obtain first page rankings for a particular site, and thus the site and its business model have to be a part of the equation. As more SEOs understand that their job does not stop when the rankings start, this will become more and more important.

The customer IMO really does not care about the rankings per se (example will they care much about a #1 ranking in mylilsearchengine.com?) but about the business results they can derive from that ranking, and now we start to get into the realm of copywriting, color coodination,ease of navigation, etc etc, which are often negelected aspects of SEO that seem to make the SEM tag more appropriate.

[orion]

You mean this EF Ratio (http://www.miislita.com/semantics/c-index-4.html).
However, I am not interested in the probability represented by this ratio. I don't use the ratio as a predictor of anything.
The EF Ratio is not a predictor of such difficulty.Wrong twice. If you only have attended SES NY...

The EF ratio measures precisely what you just described using precisely your very same arguments. So I don't think there is a disagreement here, except for the interpretion of the term "probability".

In fact the very same example (real estate) was analyzed at my "virtual" presentation on EF-ratios and c-indices at SES NY. In the case of EF-Ratios indeed these can be used to estimate how easy or difficult would be to rank for a given sequence. Several diagrams were provided at SES NY conference. They are called EF-Ratio Maps.

EF-ratios also can be used to identify the best candidate sequence from a pool of sequences and to identify natural/unnatural phrases. I might present more at other conferences/univs I have been invited to.

In the last slide "PUSHING THE ENVELOPE" I show sub sets within the EXACT mode and how the implementation can be used in optimization strategies.

Orion

[orion]

I am saying that your theoretical approach may be fine in the lab, but doesn't cut the mustard in everyday use... Let me see:

1. Dan Thies, a pioneer and icon in keyword research spent few good minutes at his SES NY presentation quoting my research work on EF-Ratios, C-indices and semantics in general and the merits of these metrics.

2. Christine Churchill, randfish, Bill (bragadochio) and others can see many values and applications.

3. Many seo/sem firms that approached me are happy implementing these in the real world with real cases and real clients in a real manner. Real really.

Still doesn't cut the mustard in everyday use?

Cheers

Orion

PS. Mel, I apologize. When I was typing this post, I made an honest mistake and edited your post I'm quoting. Please accept my apologies in advance. (Dr. E. Garcia)

[Michael Martinez]

Wrong twice. If you only have attended SES NY...

The EF ratio measures precisely what you just described using precisely your very same arguments.

Not according to what is actually being written about it, both here and on that site I linked to.

...So I don't think there is a disagreement here, except for the interpretion of the term "probability".

Wrong again. I am not interested in predicting any probabilities. You're mistaking my purposes and intents for yours, and mine are very different from yours.

[orion]

Not according to what is actually being written about it, both here and on that site I linked to.



Wrong again. I am not interested in predicting any probabilities. You're mistaking my purposes and intents for yours, and mine are very different from yours.Sir, if you have attended the SES NY conference you would have a better notion of what the EF-ratio actually measures or does not measure, predicts or does not predict.

Orion

[Michael Martinez]

Sir, if you have attended the conference you would have a better notion of what the EF-ratio actually measures or does not measure, predict or does not predict.

Orion

I am not interested in your application of the EF Ratio. I am not interested in what you measure with it. I am not interested in what you do or do not predict with it.

We are both using the same data to accomplish different results, for different purposes.

Just because you presented the EF Ratio at a conference doesn't mean I am obligated to use the base data your way.

[orion]

Certainly, you are not obligued to use EF-Ratios or any other metric, nor we intent or expect you to do so. I simply am explaining for the benefit of the fine posters of SEWF how the proposed metric can be used to tests which term sequence are easier/harder to rank for using the EF-Ratio metric. Examples were presented with several graphics at the NY conference, no more no less. BTW no prediction was involved. These were real cases.


Orion

[Mel]

Certainly, you are not obligued to use EF-Ratios or any other metric, nor we intent or expect you to do so. I simply am explaining for the benefit of the fine posters of SEWF how the proposed metric can be used to tests which term sequence are easier/harder to rank for using the EF-Ratio metric. Examples were presented with several graphics at the NY conference, no more no less. BTW no prediction was involved. These were real cases.


Orion

Orion
Did your real cases show the results of optimization on the terms selected in that fashion and the resulting traffic?

[orion]

Sure. Try mexican food, or to be unbiased, check cases with third parties such as the keyword research firms that find the metric so useful (Thies, Churchill, Bill S., others).

On other matters, I have uploaded new information on EF-Ratios. This may shed some light on

(a) EXACT mode implementations
(b) EF-Ratios
(c) keyword metric research approaches

It is available at http://www.miislita.com/fractals/overlapping-patterns.html

This is the latest material of the series of articles on The Fractal Nature of Semantics I'm writing.

Cheers

Orion

[Mel]

It seems to me that what determines the competitiveness of a keyphrase is not the millions of pages which do not rank for the search term and which the search engines do not even bother to display, but the actual pages that you are in competition with.

If you want a top five ranking in Google for term search engine optimization then there are only five pages that you have to beat to achieve that ranking; if you want a first page ranking in Google for the same term there are only ten pages in actual competition for that term and position.

Logic and experience tells me that any calculation which takes into account any than these pages is not going to be as precise as analysis of these pages alone would be and thus the emphasis should be IMO on how to better automate the analysis of those top five or ten pages.

I am sure that the E/F ratio may give an excellent indication of the general state of optimization and hence the general competitiveness of the entire field of pages competing for a search term, but it is not going to tell me how difficult its going to be to beat SEO inc out of the second place ranking for search engine optimization that it has held onto for so long.

[orion]

I am sure that the E/F ratio may give an excellent indication of the general state of optimization and hence the general competitiveness of the entire field of pages competing for a search term,...Certainly.

...but it is not going to tell me how difficult its going to be to beat SEO inc out of the second place ranking for search engine optimization that it has held onto for so long.I must agree with you on this, Mel. The EF-Ratio metric is not going to tell you how to beat SEO, inc out of the second place ranking.

On the other hand, I presented at SES, NY the very same example you mentioned but using k1 = "search engine" and k2 = optimization. Here "search engine" is treated as a single keyword and c-index is above the 100 mark in Google. The EF-Ratio is computed using search-engine optimization for FINDALL and "search-engine optimization". Hyphenation forces the system to recognize search engine as one word. The EF ratio is a high number. All this tells me that we are dealing with a very competitive query sequence.

So, understanding the degree of difficulty when semantically speaking one treat a 3-query case as a 2-query case helps to understand degrees of difficulty (however, there are some pitfalls and drawback when making the semantic reduction of the case, and the reduction technique itself -from a 3 to a 2 case- not always work).


On other matters, since I introduced the c-index and EF-ratio metrics there have been some reactionary SEOs misunderstanding what a scientific approach can or cannot do. Those reactions give me a flashback of those at the WebProWorld Forums that were misleaded by an editor into error. The readers, evidently without checking the editor's error rushed into conclusions only to later see the very same editor making a rectification of the facts. Check here http://www.webpronews.com/insiderreports/searchinsider/wpn-49-20040604FormulaForKeywordConnectivity.html and here http://www.webproworld.com/viewtopic.php?t=21161

NOTE: I was yesterday talking with Professor Brian Davison, Chair of AIRWeb, a workshop session of the W3C to be held in Japan in May, 2005. The goal of AIRWeb and Brian is to trying to reduce the gap between SEOs and IR folks. I mentioned to him I'm all for that and that's why I post and moderate the SEWF. I know that part of this endeveour consists in receiving fire from both sides, SEOs and IRs, but that's fine with me. If I succeed at reducing the knowledge divide between SEOs and IR colleagues I may have accomplished something positive.

Moderation Note: Anything to do with this topic was split to a new thread called Reducing the Gap - Information Retrieval (IR) and Search Engine Optimization (SEO) (http://forums.searchenginewatch.com/showthread.php?t=4651).

Cheers


Orion

[DanThies]

On other matters, since I introduced the c-index and EF-ratio metrics there have been some reactionary SEOs misunderstanding what a scientific approach can or cannot do.
It's not the easiest thing to understand! ;)

A lot of folks want a copy of the search engine's algorithm, or to reverse engineer it. One of the things you've shown us is that a copy of the algorithm, by itself, won't give you "the answers." You also need access to the database. What's interesting in your work is to see how much we can get out of simply having access to the database through public search functions.

[xan]

Hi Orion,

I know you presented at that conference and everything, and i'm sure it was interesting, but I don't get the whole EF thing.
I agree with Mel really. I don't see the use of applying it as it won't give any better an idea than any other method.

[orion]

Hi Orion,

I know you presented at that conference and everything, and i'm sure it was interesting, but I don't get the whole EF thing.Sorry to here this, xan. For unbiased references, you may want to ask around those marketing firms that are actually using the metric.


I agree with Mel really.
I don't.

Cheers

Orion

[PhilC]

Just seen this thread. There are some long and detailed posts in it which I haven't read, so I'll just stick my 2c in, sort of out of the blue, and reply to the orginal question, "What is Keyword Competitiveness?"

Competitiveness has nothing at all to do with the number of results returned for a particular searchterm. A search on the word 'the' will show why I think that way. A phrase is only competitive if people are actually competing to rank highly for it, regardless of how many pages contain the the phrase. That's what "keyword competitiveness" is, imo. The only thing that makes a phrase competitive is active competition.

Just as a search on 'the' returns around 8 billion results in Google, and yet nobody is competing for it, it is theoretically possible that there are only 11 results returned for a particular searchterm, but all 11 of them are competing tooth and nail to avoid being the one that is not on the first page. Competitiveness is about active competition, and nothing to do with the number of results.

An examination of the high ranking pages, and of some not so high ranking pages, together with PPC bid levels, can give very good indications of the competitiveness of a searchterm.

[NFFC]

>Competitiveness has nothing at all to do with the number of results returned for a particular searchterm

Is the correct answer.

In the real world the numbers mean zero and imho using them can hurt the descision making process, in much the same way that wordtraker and overture "traffic" suggestions do on the keyword selection side.

My handy dandy keyword competitiveness guide is this:

1. Is there any money in the term -
PPC bids can be a good guide to this. I look less at the level of the bid and more at its spread, how many advertisers and how the bids fall. For example $10 $1 and a million bids at 10c may present less challenges than a $1 99c 98c 97c 97c 97c 89c profile.

2. How many sites are a given -
Look at the top 10 [which is where we want to appear] and indentify any sites that have inherient advantadges that will be extremely difficult to overcome. If you want to rank for "bbc" then you are safe to assume that their are only 9 spots available for you to aim at not 10.

3. How many sites know what they are doing -
You either know or you don't [or like me think you know and maybe don't :)], a run through the other sites should soon pick out those who will be tough competitors.

4. How many kids in a basement -
They may not work logically but the shear unpredictability of the lone wolf SEO can produce a most worthy opponent.

To sum up - Is there money in the term? many sites that can't be beat? many sites that know what their doing? many sites with nothing to lose? Four yes's and you could be in for a long, hard fight.

[Nacho]

I noticed that the thread was going into topics, so I splitted them into two threads, leaving this one for the topic of "What is Keyword Competitiveness?". The other thread is called Reducing the Gap - Information Retrieval (IR) and Search Engine Optimization (SEO) (http://forums.searchenginewatch.com/showthread.php?t=4651).

Both of them are great topics guys. Enjoy!

[lots0]

I am of the opinion, that the amount of work it takes to rank a page for a keyword is what determines if that keyword is competitive or not.

If I have to point 2500+ IBLs at a page to get it to rank for a keyword, then in my opinion, that is a competitive keyword.

[xan]

Hi Orion,

Again, this isn't a dig:

"To sum up, for an IR system or search engine a query in EXACT mode simply
means: "Find an exact match according to the system IR rules".
If I am not mistaken, this is IR. I mean this is the point. Not only for the EXACT thing but for everything.

I still believe that the experimental work you present isn't enough to make an
interesting argument. As far as I can tell these are basically some notes
on IR with some complicated words used to explain them.

I just find it an effort to read your work, you seem to be saying the
obvious in a particularly convoluted way. No nice simple claims that you are
trying to support with proper experimental results etc... Most SIGIR, TREC, SEM, IJCNN or any other prestigious repository would never allow a paper written in such a style as it is too difficult to read. You will find all papers from these conferences are backed up by experiments carried out, a theory or method explained in a clear way, a discussion on the results, a comparison to results produced by other theories and claims, and not that many would have quotes from baeza-yates really, but that would be cited as a reference.

You aren't writing for a big conference or for that arena, so references and so on aren't really of interest or that necesary, but still, writing in a clear way and documenting experiments is a basic requirement of any such work.

If this metric is correct and of interest, surely a peer review carried out by the IR community would validate this. If not, then how can it be of interest to SEO if they are using metrics which are incorrect or not useful?

We work in a lab where everybody regularly publishes for SIGIR, TREC and so on, and are used to getting tons of criticism, which is always welcome or you end up spending all your time working on something that doesn't work. That way you can get cracking on some useful and interesting stuff that you can present there. Some pretty impressive researchers work here and constantly review my work for flaws, lack of documentation, not enough experimentation to back up a theory, and so on.

I hope you don't see this as a personal attack, I just fail to see any convincing evidence for the EF-Ratio claim apart from some ideas which you have presented.

Xan

[orion]

Hi, Xan

I'm fully aware of your concerns and questions from you and your colleages. From time to time similar questions arise from both sides (IR and SEO folks). It will be an honor to write a response to these. Just give me some time. We are currently busy addressing some problems in our consulting practice.

By the way, from which university you graduated and in what year? I would also like to know more about your colleagues and research. Who are they?

Regards

Orion

[NFFC]

>this isn't a dig

Same here but....I have to say that with respect orion you have the knack of putting the simplest of concepts into the most jargon laden seemingly complex way I have had the displeasure to ever see.

Its real simple, on the www there are only so many ways to skin the cat. As regards this topic I think that holds true more than ever, throw as many formula and numbers at it as you like.....the cold stark truth is that whatever keyword you aim at if at least 10 other people know what they are doing then its competitive.

[orion]

Hi Orion,

Again, this isn't a dig:

"To sum up, for an IR system or search engine a query in EXACT mode simply
means: "Find an exact match according to the system IR rules".
If I am not mistaken, this is IR. I mean this is the point. Not only for the EXACT thing but for everything.

I still believe that the experimental work you present isn't enough to make an
interesting argument. As far as I can tell these are basically some notes
on IR with some complicated words used to explain them.

I just find it an effort to read your work, you seem to be saying the
obvious in a particularly convoluted way. No nice simple claims that you are
trying to support with proper experimental results etc... Most SIGIR, TREC, SEM, IJCNN or any other prestigious repository would never allow a paper written in such a style as it is too difficult to read. You will find all papers from these conferences are backed up by experiments carried out, a theory or method explained in a clear way, a discussion on the results, a comparison to results produced by other theories and claims, and not that many would have quotes from baeza-yates really, but that would be cited as a reference.

You aren't writing for a big conference or for that arena, so references and so on aren't really of interest or that necesary, but still, writing in a clear way and documenting experiments is a basic requirement of any such work.

If this metric is correct and of interest, surely a peer review carried out by the IR community would validate this. If not, then how can it be of interest to SEO if they are using metrics which are incorrect or not useful?

We work in a lab where everybody regularly publishes for SIGIR, TREC and so on, and are used to getting tons of criticism, which is always welcome or you end up spending all your time working on something that doesn't work. That way you can get cracking on some useful and interesting stuff that you can present there. Some pretty impressive researchers work here and constantly review my work for flaws, lack of documentation, not enough experimentation to back up a theory, and so on.

I hope you don't see this as a personal attack, I just fail to see any convincing evidence for the EF-Ratio claim apart from some ideas which you have presented.

Xan
A two part rebuttal of this post is given here http://forums.searchenginewatch.com/showthread.php?p=39609#post39609