Simplex Optimization & Multivariate Testing in Marketing Research

[orion]

Simplex Optimization & Multivariate Testing in Marketing Research


Motivations

For years I've preaching why is so important for marketers to get educated and incorporate the essences of IR, Semantics, and SIMPLEX Optimization. (http://search.yahoo.com/search?ei=ut...x+optimization)

For one reason or the other I haven't had the opportunity of discussing in details Sequential Simplex Optimization Strategies, which is part of a more general field called Multivariate Experiment Designs, though I mentioned at SES, NY last year.

The work done by http://www.marketingexperiments.com and http://www.offermattica.com sites might help the marketing community to realize the importance of these techniques. Recent articles from these sites comparing AB Tests vs. Multivariate Testing illustrates the point.

AB, Taguchi, Factorial Designs, Simplex Optimization, etc, is nothing new. These techniques have been used for years. At least I used them back in the 80's and presented grad seminars on these at ASU. It so happen that this community has not been fully exposed to these basic techniques of Operation Research.

In a nutshell, the goal of multivariate optimization is to achieve an optimum condition within a feasible region of a system by analyzing multiple changes. This is by far a better approach than changing one variable at a time or checking one change at a time, which may mask interactions between variables or worse make one to take a local optimum for a global optimum.

I want to start this thread so marketers will start seeing the benefits of using scientific techniques in their rutinary analyses, especially with web analytics. I might cover in-depth the hard stuff at my site.

Some Credits First

So what all this has to do with marketing, site optimization, ROI, etc? A LOT!!!!!

Before delving into these techniques, credits and references are well deserved.

Dr. Flint McGlaughlin
http://www.marketingexperiments.com/see/1107

Matthew Roche
http://www.offermatica.com
http://www.offermatica.com/resources-1.1.html

Please check these and references at the bottom of this post, read them, and feel free to comment. Perhaps some serious marketers finally realize the importance of introducing scientific marketing optimization in their SEO/SEM mix.


AB vs Multivariate Optimization

What is the difference between AB testing and multivariative and how it can be used for SEO? Dr. McGlaughlin put it in simple terms in his report.

"A multivariable test transcends the limitations of a simple A/B test in two ways:

1. You can test multiple page variables at once.

2. You can use more than two variations for each variable.

With A/B split testing, it can become tedious to isolate and
optimize all of the elements on a page, one at a time. In
response, a number of companies have developed testing
platforms that will simultaneously test a variety of elements
such as graphics, background color, headline text, body copy,
"call to action," and other page constituents. Each of these
elements or variables can be tested with two or more
variations. The testing software splits incoming traffic
among the variations and shows each visitor only one version
of the composite page."


Currently there are software packages for multivariate testing. A great package mentioned in the report is provided by Vertster. If you are a process or experiment design engineer, you are probably familiar with many others that predate this one. However, this one is good enough.

To get the complete report on Multivariate Testing you need to suscribe to marketingexperiments.com The article is an eye opener, so as Matthew Roche's great site, offermatica.com

Read the report or visit matt's site

You will learn when to use AB tests and when to use multivariate testing. Both approaches have some drawbacks and workarounds. In next post I will explain in basic terms simplex optimization methods. Examples and case studies will be great for a SES presentation (New York?)


References

Taguchi Methods
http://en.wikipedia.org/wiki/Genichi_Taguchi

Articles:

Multi Variable Testing: Web Site Testing Beyond the A/B Test
http://www.offermatica.com/resources-1.1.1.html

Landing Page Optimization Using the Taguchi Method
http://www.offermatica.com/resources-1.1.2.html

Landing Page Testing Best Practices
http://www.offermatica.com/resources-1.1.3.html

Scientific Web Site Optimization using A/B Split Testing,
Multivariable Testing, and The Taguchi Method
http://www.offermatica.com/resources-1.1.4.html

A/B Testing for the Mathematically Disinclined
http://clickz.com/experts/crm/traffi...le.php/3349901

Split Testing Ad Copy and Landing Pages
http://www.webmasterworld.com/forum81/4502.htm

How to Figure out What Price to Charge via Split Testing
http://www.sitepoint.com/blog-post-view.php?id=204711

How Doing the Splits Can Be Good for Your Website Conversion
Rate
http://www.conversionchronicles.com/page.php?PageID=47

Conversion Chronicles
http://www.conversionchronicles.com/

The Power of Split Testing
http://www.marketingprofs.com/previe...4/jackson4.asp

Introduction to A/B Split Testing
http://www.wilsonweb.com/wct7/splittest_intro.htm

Vertster Blog
http://www.vertster.com/blog/


Orion

[traian]

Well Orion,

The more I read the more I am fascinate about these things.

One questions rise from here:

From wich point of view is better to start the optimization proces: marketing point of view(A/B testing, Multivariate testing) or from SEO point of view(text content optimizationand and all other SEO techniques).
Start with SEO and then you need to modify the headlines from a marketing point of view to have a good CTR. Or, create a highly marketed H1 tag but this time will not help for SEO.
If you take in account the eye tracking optimization then, move the photo from right to left and you will have another CTR...ouupss

Cheers,
T

[orion]

Hi, Train

Good question.

Perhaps this help: http://clickz.com/experts/crm/traffi...le.php/3349901

SEO or SEM? Depends on what you are trying to accomplish.

I would stay away from looking in either white or black or trying the one-size fits all. Optimizers -I'm using the term not to mean SEO or SEM but a QC or production tech- should not view these techniques as mere number crushing tools.

At one point I worked in the QC lab of a manufacturing plant. One day the batch was retested and we found the levels of humidity out of place. The EVOP (evolutionary operation) said one thing and the SIMPLEX suggested another as a course of action. The problem. We had to stop the batch. The office manager said: "either will cost x amount". Guess who the plant owner listened?"

So, again it depends what your client and you want to accomplish.

Orion

[traian]

Originally Posted by orion

Guess who the plant owner listened?"

A third party linke in one of my case

[randfish]

Dr. Garcia,

I know you've explained these techniques privately to me as to how they apply to thing like the SEOmoz tools. Would you share with the community some solid examples of how this type of research and information could be used by the everyday SEO or marketer?

In other words, if an SEO campaign for a website wanted to take advantage of these resources, what would they do?

[traian]

That will be great indeed

The A/B Testing and Taguchi Method are used to test the user behavior related to visual elements on the page.

The SEO tests are a lot like A/B Testing(change one element and then wait to see the results) only that the results are visible in ranking changing not on CTR changes. Also the A/B testing for SEO can be conducted in parallel, to check different SEO factors, but the problem is that the tested factors separately can have good effect but when you test the same elements together you won't achieve the ranking espected.

I am aware that are ways to test(and track) multiple factors at the same time, with Taguchi method, but I haven't yet applied.

Thanks,
T

[claus]

So, nobody's doing multivariate analysis out there? He. I think there might be a few

I agree 100% that the majority don't even know (or need to know) what it is but to extrapolate to the whole community is not entirely right.

My personal background covers the whole curriculum from Marketing Research, Operations Analysis, (even Industrial Organisation with Game Theory and such), to "pure statistics".

One thing I can say with a very high degree of certainty is that most people in this industry will never need any of this. Those that will, will increasingly find themselves employed either in toolmaking, in research, or in marketing. Not SEO.

And why do I make that blanket statement? Well, because brute force multivariate testing of Google SERPS position isn't likely to produce anything interesting, even if you combine with time series analysis to compensate for lags. Why? Okay, for starters because you don't have, and never will have, the full data set.

Yes, that in itself is of minor importance if you can make a representative sample, but can you do that? Uhm.. I doubt. And, even if you could you would need to know which exact variables to track, but do you know that with any degree of certainty? Not really, not even most of them. And for those that you do know about, can you get realiable data that looks anything like the data Google has? Not at all.

But, can you run tests? Sure! Can you get data of some kind? No problem at all. Can you crunch some numbers? Yes of course! Even produce pretty graphs and draw conclusions? Piece of cake! But would it be misleading? You bet!

So, for SEO, forget it.

For marketing, OTOH, there's a... well, the word "tradition" could even be used... for using formal quantitative (and qualitative) techniques, and that way turning data into actionable information, and hence profit. However, the people doing that have formal educations in the matter at hand, and that is neccesary because it's not just about the numbers, it's the whole method involved in deriving those numbers and interpreting them. Any fool can produce numbers, even with advanced tools. It's like giving the famous typewriter to the monkey.

So, having a marketing background I'm glad that you introduce the people here to some of the interesting stuff market researchers do, but I'm sad at the same time because there's just no place in this context to tell the whole story.

One can always hope that the people interested find occasion and economy to invest in getting and reading some relevant textbooks and/or taking some courses, but in real life they will most likely see some formula, and apply it non-critically to any problem they might face. After which they will turn to "the expert" and say - "that stuff you told me didn't work". Rightfully so, because they have not had the full introduction, not even the patience to get it.

Anyway, I should shut up and get on with life. So I will.

I just intended to leave a small note of caution to those looking for "the holy grail in SEO", that "there are more fruitful ways to get your site ranking - especially developing your site"

Apart from that it's nice that you introduce these concepts to the community Orion. I'm just worried that the readers will make all kinds of wrong decisions. These are strong tools and they do need strong disclaimers.

[orion]

Thank you for your feedback. I havent' being able to follow up this thread due to lack of time. I hoped this would give time to others to throw in some comments.

Here is my reply to you, claus.

I agree 100% that the majority don't even know (or need to know) what it is but to extrapolate to the whole community is not entirely right.

I agree, but nobody is extrapolating anything. My comments appealing to education stands.

The problem with the rest of your post and rest of reasoning is that you assume this has been suggested to SERPs. Nobody is suggesting that. Still, I know of some optimizers that have tried to do this in the past.

Multivariate testing is an umbrella term. Simplex optimization (not to be confused with simplex linear programming) is a specific sequential optimization technique used for many years in marketing and process improvement. I've done all that prior the Internet.

Simplex, in particular is used to make educated decision, not to replace how-to knowledge or to invoke any "holy grail". No need to invoke to pity, either.

I agree with you that people makes all sort of stupid decision for any sort of stupid reasons and say a lot of stupid things.


Orion


PS. Claus, when replying I by mistake edited your post. I apologize. I put back your text.

[claus]

Oh, well, the context is a Search Engine forum populated by a membership of mainly SEO / SEM people. That, by itself, suggests the application to SERPS. No human intervention needed.

Some of these people think only in terms of ranking, that's all I'm saying. I'd even say that that should be obvious after reading some earlier threads on various research subjects Whatever they see they apply to this specific problem. Tell them about any scientific theory (eg. gravity) and some of them will conclude that that theory must be important to their rankings. Only because they read it here. Similar to any kind of technique. Hence the need for explicit disclaimers.

There's a subset of marketing and/or research people here that understand some of these topics a bit better and find them interesting "as is", but we (I count myself among them) are the minority, unfortunately.

(And I don't care much for grails myself, of course. That was just a metaphor for the extreme focus on rankings.)

[orion]

Originally Posted by claus

Oh, well, the context is a Search Engine forum populated by a membership of mainly SEO / SEM people. That, by itself, suggests the application to SERPS. No human intervention needed. Some of these people think only in terms of ranking, that's all I'm saying.

I disagree with this perception. Some SEWF posters and threads discuss things unrelated to SERPs or rankings. Some are about strategies urelated to SERPs, but about things like how to improve ROI, how to cut operational costs, software beta testing/reviews, event and news announcements, crawling techniques, etc, etc. This section, "Search Technology and Relevancy" has many threads that are not necesarily about SERPs or rankings at all. Some posters only care about revenue strategies, not rankings. Even others use this forum for networking.

Having said that, I agree with you that there are some ranking-obsessed folks that only think in auto-pilot and that only value a theory or thread using their little SERP mentality.


Orion

[claus]

Well sometimes I should just keep my big mouth shut. It wasn't nice of me to make a blanket statement about the audience here. If I offended anyone I'm very sorry, my apologies for that. Back on topic..

[Montgomery88]

My two-cents about using multivariate testing or A/B testing to "game" the SERPS is that is just isn't practical. Even if you ignore the potential risk of being banned altogether for rotating content for grey/black SEO, there are just too many uncontrollable/unknown variables. Consider that SERPS change from day to day even when one's site remains the same -- how would you reliably infer that a particular change you made to your site caused a change to your SERPS? The unreliability of this approach simply increases the the number of changes (combinations).

SEO aside, A/B and multivariate testing are certainly useful for optimizing visitors once they're on your site. I think this is an approach that is here to stay, and hence there are several companies with dedicated testing products/services. Another good reference (product site) to investigate is http://www.sitespect.com. While they provide testing and optimization ala Vertster and Offermatica, SiteSpect appears to do this without the use of JavaScript. SEO's should appreciate the benefit of not having to add more JS or tracking codes to their site.

greetings,
Monty

[orion]

Thank you for posting your comments, Monty.

Originally Posted by monty

My two-cents about using multivariate testing or A/B testing to "game" the SERPS is that is just isn't practical...

I believe I addressed this in my reply to Claus. See post #8.

As mentioned, multivariate testing is an umbrella term that covers many techniques: factorial designs, fractional factorial designs, simplex optimization, EVOP, etc... Some authors consider A/B testing part of mutlivariate testing.

This thread is about sequential simplex optimization (not to be confused with simplex linear programming), one method of multivariate testing. The optimization technique has been applied for over 40 years now in many fields such as marketing research, economics and even chemistry.

For instance if I want to improve the yield of a chemical reaction in a process by changing at the same time sequentially two variables, say, temperature and pressure, I can find the optimum T and P conditions with simplex optimization. If I want to optimize the cost of a product by changing X or Y and Z, etc variables, I can do this with simplex. The same applies to other controllable variables in other problems and disciplines...


In general, multivariate testing of any sort performs poorly if the results do not scale or are not reproducible up to a certain degree between laboratories.

Non of these methods can be safely applied to SERPs since with search results one deals with things that escape to reproducibility/scalability; e.g., to mention a few semantics, perception of relevancy (from a user or machine perspective), and the ever changing recipes that search engines cook under the hood.

Orion

[Montgomery88]

Originally Posted by orion

Non of these methods can be safely applied to SERPs since with search results one deals with things that escape to reproducibility/scalability; e.g., to mention a few semantics, perception of relevancy (from a user or machine perspective), and the ever changing recipes that search engines cook under the hood.

That's it, exactly. There are so many variables going into the SERPS, it's even been suggested that there are random-like components to their [e.g. Google's] algorithm. Even with "lurking variables", as statisticians like to say, at least you can re-run an experiment and try to quantify/control for them. With SERPS, however, those variables could be constantly changing as their algorithms evole and G tries to stay one step ahead of SEO's. That puts those variables into another problem category altogether: unknown unknowns.

M

[orion]

As mentioned before, this thread is about sequential simplex optimization. For those interested in AB testing, there is a practical and a bit superficial description at this old SEWF thread: AB testing of creatives


Orion

[seomark]

Hi guys, I have a question and hope you might be of help. I found this MuVar multivariate testing tool developed by James Brausch, I wonder if anyone here has tried it and if it’s worth the $100? I’m thinking of buying it, but I thought I should ask you guys first.

Thanks a lot!