Tools to analyze keyword search frequency

[wbednarz]

Are there any online tools available that allow you to find out how often a specific set of keywords are queried? I believe go.com used to have a tool that offered this tool at one time for free, and charged a fee.

[Terry Plank]

One of the standard fee-based tools is Word Tracker (http://www.wordtracker.com). You can get use for a day at $7.60 USD.

[pleeker]

And Overture offers a search term suggestion tool (http://inventory.overture.com/d/searchinventory/suggestion/) that's updated monthly with query totals on the Overture network from the prior month.

[Terry Plank]

Some other options:
Espotting (http://www.espotting.com/popups/keywordgenbox.asp)
SE Report (http://www.mall-net.com/se_report/)

[5starAffiliatePrograms]

I like this one which shows WordTracker AND Overture daily totals.

http://www.digitalpoint.com/tools/suggestion/

================================
This one calculates Google search frequency
http://ecommerce.insightin.com/tools/keyword_popularity_calculator.php

================================
This is a new tool I just found that gives tons of KW data from different sources.

* List related keyword phrases from Google™.

* List related Keywords from Overture™ with number of searches conducted.

* Number of Pages related to that given keyword or phrase in Major Search Engines like Altavista™, Yahoo™, MSN™, AllTheWeb™, AOL™, Teoma™ etc.

* List URLs of Top 5 Ranking WebSites in Google/Yahoo.

* Display Link Popularity, Meta Keywords and Alexa™ Ranking of 5 Top Ranking sites.

http://smtek.info/cgi-bin/keywordtool.cgi

Enjoy! ;-)

Linda

[Terry Plank]

Linda, thanks for the heads up on http://smtek.info/cgi-bin/keywordtool.cgi. For Overture keyword searches, know what the number is they give is? I'm assuming it was a WordTracker 24 hour period one.

Interesting to see the Yahoo! Directory number of pages and link results comparing to Altavista, MSN, All The Web, AOL and Teoma.

[DanThies]

This one calculates Google search frequency
http://ecommerce.insightin.com/tools/keyword_popularity_calculator.php

I wouldn't put too much faith in this one. For terms where we are advertising on Adwords, the 'estimate' provided by this thing is typically off by an order of magnitude.

[rustybrick]

I wouldn't put too much faith in this one. For terms where we are advertising on Adwords, the 'estimate' provided by this thing is typically off by an order of magnitude.

Dan, what would you recommend? You give a detailed presentation at SES on this, maybe you can share some goodies here. :)

[5starAffiliatePrograms]

Yes Dan, please share some goodies.

BTW, I don't claim any accuracy for any of these, I just like to share whatever tools I find in case they are helpful to someone.

Linda

[DanThies]

I think Wordtracker is still the best data source for organic search. Not perfect, but far better than the alternatives. We do our research with Wordtracker data, and add additional data from other sources to round out the picture.

A step up from there, once you have selected some search terms, is running an Adwords campaign. As long as your click-through rate and bid are high enough, and you don't underbudget, you can get real search counts from Google.

An Adwords campaign will also give you a very good idea how relevant your chosen terms are to searchers. If the click-through rate for a given search term is extremely low, it may not be the best candidate. (Article: Find Your Best Search Terms With Adwords (http://www.webpronews.com/2003/0708.html)). With geographic targeting, Adwords can also be used to determine the popularity of search terms within specific countries or regions.

I would note that you can't get the same information from Overture. The main reason being that they don't let you opt out of their affiliate network. So a "search" might consist of a pop-under window generated by GATOR, metasearch results powered by Hotbar or Toptext hijacking content, click-throughs to expired domains, etc. This skews all of the data you get from Overture, whether free or paid.

I'm seeing a lot of 'keyword tools' popping up these days, but I'm still waiting for something new and useful. One that's been around for a while, and occasionally helps with keyword discovery, is Wordnet, (http://www.cogsci.princeton.edu/~wn/) which can be used online (http://www.cogsci.princeton.edu/cgi-bin/webwn).

We're going to be releasing a few of our keyword research/discovery tools as open source in conjunction with the San Jose SES conference. Where, BTW, we now have an "advanced" session on the 4th day in addition to our basic session on day 1. The advanced session will be great fun, where we don't have to stick to basic topics.

[Nicky]

It would be nice of some of these tools gave UK results only :( I know overture do, but not based on organic searches.

Incidently, where do WordTracker get there results from? Are the meta engines US based, or the whole of the web or maybe universe? (ok, a bit OTT there :D )

[DanThies]

Wordtracker's results come mainly from Dogpile and Metacrawler. Overture does have a UK specific tool, but when it comes to keyword tools, an SEO in the UK basically gets little help.

Keyword discovery is one thing. Once you have a list, an Adwords campaign can tell you how many people are searching for it, with whatever geographic focus you like.

[dejaone]

I wouldn't put too much faith in this one. For terms where we are advertising on Adwords, the 'estimate' provided by this thing is typically off by an order of magnitude.

I did some major enhancements on the tool. I'd say it's fairly accurate now. Try it again.

http://ecommerce.insightin.com/tools/keyword_popularity_calculator.php

I've implemented a web-based wordnet already (not for SEO purpose). I plan to integrate the keyword tool with wordnet. Any suggestions would be welcome.

[Nacho]

I did some major enhancements on the tool. I'd say it's fairly accurate now. Try it again.
I made a few tests and unfortunately they are off by over 90%. I also get a "runtime error" on line 11.

I hope you can get it going, I would love using it on a frequent basis if it can be reliable. You're doing a great thing, please keep tweeking it. :)

[webconnoisseur]

I'd definately say the tools are far from perfect. I like wordtracker, but their analysis on what words to go for is way off base, sometimes. There are many other factors that should be taken into account when trying to optimize for organic results (like how well optimized the competing pages are). I usually build a spreadsheet with my terms, then rate the words on a few different factors I find important, then apply my own algorithm to determine the words to go after.

I will also follow a similar process (with a different algorithm - more focused on the costs) when choosing my ppc keywords and bids.

[DanThies]

Yeah, don't count on KEI. We take a very simple approach, multiply the search volume by an estimate of the term's relevance, to get a "weighted popularity." Don't want to bore the group with mechanical methods of estimating relevance, because I don't think they're any better than an educated guess, although we're working on it.

[northwest]

I use Good Keywords (goodkeywords.com). Pretty simple tool... Has anyone tried this??

[5starAffiliatePrograms]

Yes Good Keywords is one of my favorites. It does so many things quickly and easily. Much better than going to Overture and it saves your searches.

[DanThies]

Yeah, GKW is great - we use it every day. I don't know why anyone would use the Overture tool online, if they can run Good Keywords instead. It's a great way to add words and concepts to our seed lists before we dig into the real nitty-gritty keyword research work.

[orion]

Hello everyone.

Fair and good approach, Dan. I remember back in 98-99 that the NEC Corporation tried to implement a system in which keyword search frequency, click-through rates, and keyword weights were combined in a double sum model. The idea consisted in connecting the dots between keywords search frequency and advertisement.

It was supposed to be a self-updating keyword-based learning system. It was called ADWIZ. Now part of NEC's CRM services, it has been integrated in their VS-7820 system (http://www.nec-globalnet.com/products/vs7820.html)

For those interested in tweaking a keyword model, here is the original NEC implementation of ADWIZ.

http://www.vs.inf.ethz.ch/publ/slides/adwiz-www8.pdf
http://www8.org/w8-papers/2b-customizing/unintrusive/unintrusive.html

The original math apparatus was essentially correct, but it missed latent semantic and ontology components such as co-occurrence, sequencing, ontology building blocks (N-N, noun-noun, A-N adjective-noun, etc) components. These elements should not be overlooked by candidate keyword research tools.

It appears NEC has improved the VS-7820 system since then.


Orion

[DanThies]

Since the cat will be out of the bag shortly, one of the tools we're releasing at SES is based on the information that Orion has provided in another thread, and on his website.

This tool takes a search term as input. For example, "auto insurance." Then it queries Google for the top 10 results (using the API), and requests the cached copy of each one. Once we have the top 10 pages (or whatever subset is available), we mash 'em all together into one big file, and use one of the functions from our keyword density analyzer to produce a list of 1, 2, and 3 word search terms from the combined pages.

What you end up with, then, is a really huge list of words and phrases. Only some of them are really search terms, others are just words that happened to appear in sequence. This is where the information Orion provided becomes extremely useful.

The first tool we have in our bag, courtesy of Orion, is the "E/F ratio." (Long technical explanation here (http://www.miislita.com/semantics/c-index-4.html).) This compares the number of times the words appear in an exact phrase match vs. in a "find all" mode. So we take our list of 3-word phrases, for example, and query Google twice for each one, once with quotes around it and once without, to get an E/F ratio. The higher the E/F ratio, the more likely this is an actual search term, naturally occuring in the language.

We can now add color coding to our list of candidate search terms, to highlight the most likely search terms, based on the E/F ratio and the number of times each phrase occured in the pages we fetched from Google. The user then selects from the candidate search terms with checkboxes, those search terms that actually seem that they may be useful.

Now we have a list of good candidate search terms, but we aren't done yet. We then take this list, and calculate c-indexes with the initial search term ("auto insurance"), to provide some insight into which are more relevant. For this calculation, we use exact phrase matching only. The final listing is sorted by c-index scores.

We will release the PHP source code for this application on August 5, in conjunction with the SES Advanced Search Term Analysis session.

What we won't release immediately is the third part of that application, which communicates with the Wordtracker API to get search counts for the candidate search terms, and extract additional related terms. It's not really ready for prime time yet, and I don't believe Wordtracker is selling their API yet in any case.

Further development will involve using c-indexes and other methods to look for semantic relationships between search terms, to assist us in grouping them by theme, more or less automatically.

[rustybrick]

Dan, that sounds awesome!

[orion]

I'm very happy seeing Dan applying the basics of the c-index and EF ratios. Excellent! This simplifies my work trying to convince SEO and SEM specialists of the merits of the theories I have presented. Awesome.

I invite others to follow into Dan's steps and welcome this and similar tools. We have an expert system that conducts the essence of Dan's tool in a TVT/LSI framework. If we can help to co-organize a SES in San Juan by 2005 we could introduce the system to fellow SEOs/SEMs.

Orion

[DanThies]

Orion,

I haven't really found any applications yet to the many basic practices of SEO (such as optimizing page design and copywriting), but these concepts, that you've taken so much time to explain, have reaped tremendous benefits within our process of keyword discovery. Not only does it allow us to be more thorough, it offers the possibility of insight into the true relevance of search terms.

The end result of this application development process will be better tools for keyword researchers, and a greater emphasis on relevance and targeting from the SEO firms we work with.

Even if folks don't use the tools the same way we do, there will be an open source PHP class library supporting further work on c-indexes, E/F ratios, etc. A better understanding of the theory underlying information retrieval and search benefits us all in the long run.

[orion]

Dan

Thank you for such kind words. I wish one day we can meet in Texas or if Jupitermedia and SEW ever make it possible, at a SES in beautiful San Juan, Puerto Rico.

1. "Not only does it allow us to be more thorough, it offers the possibility of insight into the true relevance of search terms."

Yes. This is the essence of what I'm trying to show to SEOs; ie. the true meaning of relevance, not just a mechanical definition of it.

2. "The end result of this application development process will be better tools for keyword researchers, and a greater emphasis on relevance and targeting from the SEO firms we work with."

I'm all for that. The more keyword researchers jump in the vanwaggon, the better for all us. The main thesis of all my posts at the SEW threads is to show SEOs that optimization is not about trial-and-error approaches. Systems in which many variables are involve can be dissected and optimized using many analytical tools (simplex optimization, factorial designs, evop, etc) without the need for second-guessing things. I will stop here since I don't want to give the wrong impression; ie., that I know everything or that I'm trying to lecture now. Let's just pass the word about this work and let others make a decision.

3. "Even if folks don't use the tools the same way we do, there will be an open source PHP class library supporting further work on c-indexes, EF ratios, etc. A better understanding of the theory underlying information retrieval and search benefits us all in the long run."

Great, Dan. Let's just pass the word about this work and let these folks draw their own conclusions. Hopefully they can see the benefits of such approach. You have made my day.

Even if average SEOs/SEMs cannot see the benefits derived from c-indices and EF ratios, I wish to see the day where the "meat" of IR concepts are regular part of search engine conferences. Put me on the list. I'm always willing to cooperate.


Orion

[DanThies]

David Warmuz of Trellian recently posted some info on another forum (http://www.highrankings.com/forum/index.php?showtopic=7826&st=0&#entry84385) about a new keyword research tool they've deployed within their Priority Submit service: http://www.prioritysubmit.com/research.html

I don't know if it's been announced here, but it looks promising - the paid inclusion programs have always been a vast repository of keyword data.

[trellian]

Thanks Dan,

David Warmuz of Trellian recently posted some info on another forum about a new keyword research tool they've deployed within their Priority Submit service: http://www.prioritysubmit.com/research.html

Here is a list of key features:

- free (currently)
- 9 billion search term database
- obtained from close to 40 engines (ppc, international,meta and regional)
- provides top 1000 results
- provides related search terms
- search term analysis
- Keyword Density & Cross Referencing
- Regional keyword research

http://www.prioritysubmit.com/research.html

Always appreciate any thoughts and feedback!

Cheers
David

[Terry Plank]

David, nice to have you posting in the forums. I see in your post at the other Forum that you will disclose more later about what sources you are using for your database, I'll look forward to learning that.

In the mean time, I'd be interested in the thinking behind developing this tool. Certainly not asking you to specifcally evaluate others, just your own rationale for spending the time of developing this tool and how you think it will be helpful to add to our tool kit. How might it be used strategically that would add value to what's currently available?

[DanThies]

Terry,

The #1 feature of this system is the one that David isn't touting, which is that they have a full year's worth of data. You can go see how many searches there were for "chocolate bunny rabbit" back in March, and in fact get a cool little chart showing the month by month trend.

Seasonality is the biggest weakness of the other tools - you can't go back in time and see what happened last October with Wordtracker, Overture, etc. Now you can, with Trellian's tool.

It looks just as skewed as anything else, in terms of the effect of rank checking, and I'd still say Wordtracker's database might be a little cleaner from that perspective, but this is a tremendous addition to the tool bag, and probably destined to become the new #1 resource for keyword research.

[orion]

Trellian (David):

That's sounds great. Is there room for prospective seo/sem/buyers interested in validating the raw data?

It would be interesting to know the degree of bias introduced due to fraudulent click throughs in keyword marketing programs and how this bias plays into keyword searches and demographic data.

If Trellian or someone has already this kind of info, let us know. We're conducting a research work on click through raw data and terms co-occurrence/sequencing. The goal is to build a math model and make some predictions using real-time data (fraudulent and valid). We are interested in queries consisting of 2-3 words, only. We also want negative and control terms. We can talk through regular email.


Orion

[Terry Plank]

Dan,

Thanks for the intro to the tool here and also for the "seasonality" advantage tip.

I'll be interested to learn more eventually about the sources Trellian is using to gather data so we can get a better sense of the skewing of rank checking. Webtracker is certainly superior at the present in that sense because of using metacrawler data.

[trellian]

While still in beta the results are naturally still being refined and I would expect this to continue to remove and address any skews as they are identified.

Orion, yes you are welcome to email me directly david@....

In the mean time, I'd be interested in the thinking behind developing this tool. ...... How might it be used strategically that would add value to what's currently available?

Terry, we have know from our association with the PFI programs, PPC campaign management and our SEO Toolkit software, what importance keywords play and were simply not satisfied with any of the current tools available to suit our own needs. When this happens, we try to see if we can do something about it.

Initially this was made for our own internal use.... until some of the features were mentioned at the last 2 SES shows ... The response was amazing. Far too many SEOers had the same need.

Cheers
David

[marxes]

I have been playing around with priority submit for a day or two and I have some questions about what I am really being presented with. In particular the "searches" column that is listed when you do a keyword search. I have been taking this to be the average number of searches per ? I'm not sure what the time unit is. I think it is really convinent to be able to do the density checker and then the keyword analysis in the same tool. My only beef so far is that it is a little slow, which isn't a big deal considering that not only is it in beta, but it's also free. Keep up the good work!

-marxes

[trellian]

the "searches" column that is listed when you do a keyword search. I have been taking this to be the average number of searches per ?

this provided the total number of searches from our entire database.
It used to be based on the last months worth of searches and depending on the feedback we receive it may go back to a monthly figure.

Yes the speed is an issue, but not for long as a scheduled move to new/faster hardware is already in the works.

Cheers
David

[trellian]

Just a quick update on the KeywordDiscovery.com Keyword Research Tool (http://www.keyworddiscovery.com/) we have recently launched a new feature called Directory Keywords (http://www.keyworddiscovery.com/keyword-directory.html) that is the 1st phase of Competitive Intelligence.

What it does is identifies what specific search terms drive and send traffic to web sites within each specific category.

The primarily use for this tool is competitive intelligence when performing keyword research, as you can navigate to where your main competitors are indexed in DMOZ and identify what exact search terms are responsible for driving the most amount of traffic to all the sites within that category.


Plus if you have your own online tools, we have free API solutions for our keyword data to be integrated. Contact me for more info.

Feedback is always welcome.

Cheers
David

[HardlyNoticable]

dejaone:

Your tool (http://ecommerce.insightin.com/tools...calculator.php) is broken. I tried leaving you an email through your contact link... but that was broken too.

[jimbeetle]

Not surprigins, that post was more than three years old. Looks like the entire domain has, erm, changed a bit.