The ROI from SEM can not be measured accurately

[Reality Check]

I am in a debate where I need to prove the ROI from our SEM program is real and not just smoke and mirrors.

The problem is… I’m loosing the argument. I truly believe that cutting the budget would do much more harm than good but I can’t figure out how to prove it.

The arguments I’m up against are just too simple.

In this thread, I will play the Devils Advocate and use these simple arguments
to debate with anyone who thinks they can prove these arguments wrong. Nothing would make me happier than if someone can do just that.

First some background information to help frame the debate.

• The website is for a large brand that receives a large volume of visits and transactions

• The goal is to drive sales on the website as these direct transactions are the most profitable for the company

• SEM is only one of several dozen channels of product distribution

• There are dozens of ways to reach the website other than search or PPC

• The revenue contribution from SEM is significant but it’s only a fraction of the total volume of transactions

• Sales volume and visits can very greatly due to competitive conditions and seasonality. The typical % variance is grater than the usual contribution from SEM

• The SEM campaign is measured by a strict direct response metric for cost of advertising versus revenue generated. Branding and raising awareness is not a goal or even a consideration.

Here we go…

The ROI from SEM can not be measured accurately.

• The PPC ads for a particular keyword are either there or not.
• You can only track and measure the behavior of people who click on your ads.
• You can NOT track and measure the behavior of these people if there is no ad to click on.

If you can’t track and measure the people who were not served an ad, how do you know that these people are any les likely to make a purchase on the website?

Just because someone clicks on the ad and makes a purchase is no reason to assume that they would have not purchased the product anyway without the additional cost of the click.

Several, Many, Most or All of the people searching for a particular keyword will find the website with or without the help of PPC ads. Exactly how many of these people will reach the website and make a purchase is unknown.

If ALL of the people who click on the ad would have made a purchase on the website anyway, the ROI from the advertising expense would be Zero.

The % of people who would NOT make a purchase because the ad is NOT there to click on, is the true measure of the ROI. Because the behavior of people can not be tracked or measured without serving them an ad, the ROI can not be tracked or measured. (No control group)

If we don’t know if it’s working or not, than why are we spending millions of dollars on it?
Why not spend the money on something else or just save it?

XXXXXXXXXXXXXXXXXXXX

We also do quite a bit of Banner advertising but this is not perceived as smoke and mirrors because the ROI reporting is much more accurate and accountable than search due to the ability to measure test and control groups against each other for the same time period.

XXXXXXXXXXXXXXXXXXXX

We can run a test where we show half the audience a real banner and the other half the audience a bogus banner for a charitable organization or whatever.

Once you have been segmented into one group or the other you will continue to stay in that group. You will either see real banner ads everywhere you look or the bogus banners for the duration of the test or until you delete your cookies.

Several, Many, Most or All of the people who are only served the bogus banners will make a purchase anyway.

Let’s say this is the results…
7,000 people who only saw or clicked the bogus banners made a purchase anyway.
10,000 people who saw the real banners made a purchase.

The real banners in this example show a 3,000 lift in orders.
The ROI contribution of the banners would be attributed to 3,000 orders. (Not 10,000)

XXXXXXXXXXXXXXXXXXXX

This sort of test and control methodology is not available with search.
This is why the ROI can not be accurately measured.

Before you can accurately determine ROI from search you would need the ability to track people who search for a keyword or group of keywords and are NOT served a PPC ad. This will tell you how many of those people will make the purchase anyway or more importantly how many will not.

Google and the other search engines would need to allow advertisers to purchase a tracking pixel instead of the actual PPC ad for a % of the tested keyword queries. People in one group or the other would need to stay in that group and either see the PPC ads or not for the duration of the test or until they delete their cookies.

Until this happens, we have no way to determine if the millions of dollars spent on PPC is a wise decision or not. If we don’t know of there is any ROI or not, then why are we spending the money?

[andrewgoodman]

That was a pretty detailed post - I think the issues at stake can be boiled down pretty far, down to a couple of sentences.

Someone of good faith in the organization needs to believe that there is incremental revenue that can be created through a paid search program.

When some methodology is set up to attempt to prove that, they need to assess the information fairly and open-mindedly.

As you say, you can track clickers all you want, but if the argument becomes - "what's to say they wouldn't have bought anyway," you're fighting a rhetorical battle against someone who won't let you win the argument. It would take some pretty complicated math to *prove* that there is little chance that they would have bought otherwise.

Direct response metrics are the primary ones to consider, but there really is the cumulative effect of using all your promotional resources over time. When someone types "page zero" into a search engine or navigates directly to my site, I can't now prove that this lead was generated from any particular source. All I can do is be bullish on the value of various forms of promotion I'm using and measure them as well as I can. If I stop all promotion, guess what: no growth, fewer and fewer sales.

If the campaign is large enough, you could turn it on for a month, and then off for a month, and look at total revenues. Did you move the needle? Even here, a sceptic could chalk things up to seasonality if you were trying to dismiss the impact of paid search.

This argument could be made in connection with any form of advertising, though. Are the other methods being measured? Why would someone want proof that one method works, and not demand equal proof of the other methods' incremental value?

Some of the argument behind this may come from math proposed by analysts like Jim Novo, trying to help companies avoid "cannibalizing" their channels by overspending in PPC where they don't have to (because, let's say, they appear in organic etc.). Not all of Jim's premises hold with regard to today's search landscape, IMHO. It's haggling over nickels and dimes, usually with regard to very large companies who are quite wasteful in other areas of their advertising.

You're in a political type argument, it sounds like. Send them my book. :)

[andrewgoodman]

Just to add a bit to this, it may be possible to start debunking some of these assumptions about the likelihood of purchase from various sources, by looking deeper into analytics.

For one thing, is there a broad-brush assumption that the company is getting a whole lot of juice from organic rankings. True or untrue? What is the actual monthly profile of organic referrals (not rankings), and could this be improved? Is there a ceiling for such improvement because Google et al control the SERP's and have a new, post-Florida philosophy that pushes commercial pages down the list more often than not, let alone the onebox type results that may show local search, news search, and other prompts at the top of the page, crowding you out of view?

If a company really wants answers, they'd probably get set up with Clicktracks Pro or something else so they understand more about user behavior post-click from both paid and unpaid search. If they think search of either type has little incremental value, well... I can't help you. :) A good user experience online is part and parcel of living in "today." So what if there are flyers and storefront locations, and TV ads for direct navigation, or whatever? When people go searching, the more often they get coherent representation of a brand, the more aware they are of your company as one with a solid online presence.

Many of my clients would not be on the map at all without PPC. They simply get no orders. In larger organizations, there can always be a political argument that says, we don't need this or that. We're doing great anyway, etc. My take on these no-win arguments is generally, you can lead a horse to water. I'll focus on the people who really want to proceed with a paid search program. There is only so much you can do to sell something to someone who isn't buying. My wife and the 100 vendors who sell metal doors can give me 3,000 facts about why a metal front door is better, and I'll still wind up with a wood front door even if I need to hire a separate contractor to tear down the metal door she buys and hang my big wood door. :)

At the very least do they not understand the market research value of 2-3 months of tracked paid search campaigns on a wide variety of keywords? Wouldn't they learn more about which landing pages convert if they ran systematic campaigns? Geez.

[rustybrick]

Let me see if I can accurately summarize the issue here Reality Check.

Big Brand X has PPC ads for the the keyword phrase of the Brand Name of Big Brand X.

That keyword phrase brings in $10,000 in sales per day but costs $5,000 per day on PPC spend.

You remove the keyword phrase from the campaign and you have the same gross sales on your site.

So in essence you are saving $5,000 per day.

But the tracking tools show you have a ROI of $5,000 per day.

Is that a good example?

[AussieWebmaster]

I am in a debate where I need to prove the ROI from our SEM program is real and not just smoke and mirrors.

The problem is… I’m loosing the argument. I truly believe that cutting the budget would do much more harm than good but I can’t figure out how to prove it.

The arguments I’m up against are just too simple.

In this thread, I will play the Devils Advocate and use these simple arguments
to debate with anyone who thinks they can prove these arguments wrong. Nothing would make me happier than if someone can do just that.

First some background information to help frame the debate.

• The website is for a large brand that receives a large volume of visits and transactions

• The goal is to drive sales on the website as these direct transactions are the most profitable for the company

• SEM is only one of several dozen channels of product distribution

• There are dozens of ways to reach the website other than search or PPC

• The revenue contribution from SEM is significant but it’s only a fraction of the total volume of transactions

• Sales volume and visits can very greatly due to competitive conditions and seasonality. The typical % variance is grater than the usual contribution from SEM

• The SEM campaign is measured by a strict direct response metric for cost of advertising versus revenue generated. Branding and raising awareness is not a goal or even a consideration.

Here we go…

Okay I will see if I can add something here.

The ROI from SEM can not be measured accurately.
This is the biggest false statement made in the post. SEM is the most accurately measured online marketing method. Any caveats - and blind tests - can just as easily be used as a counter for any marekting efforts. Your guys are just using theirs first and claiming it cannot be replicated.

Any advanced web analytics program can give you the results for paid and organic conversions. They can also capture secondary or initial marketying influences prior to a sale. You then need the CRM program to have the ability to keep the information and thus get numbers for long term purchasing practices for all marketing efforts.

• The PPC ads for a particular keyword are either there or not.
Okay and if you don't pay your banner bill they don't appear - or if you do not buy all possible impressions your banner is rotated - this is an issue of camapign budget limits.

• You can only track and measure the behavior of people who click on your ads.
Not true.... if you do not have an organic listing on the page.... you can pretty much measure the fact that all searchers did not go to your website from the search page.....


• You can NOT track and measure the behavior of these people if there is no ad to click on.
Say the same thing in reverse always adds momentum against a proposition


If you can’t track and measure the people who were not served an ad, how do you know that these people are any less likely to make a purchase on the website?
Sounds like the tree falling in the forrest routine....
The first hint to this is if they are about to buy and you have competitors who are advertising - you are going to lose potential customers - this is not a vacuum - there is no simple answer but you should consider the competition in there - if you have none then why bother advertising at all.



Just because someone clicks on the ad and makes a purchase is no reason to assume that they would have not purchased the product anyway without the additional cost of the click.

All very true if you are the only provider - though alternatives would be in play if you were - but if you have competitors they get the sale!



Several, Many, Most or All of the people searching for a particular keyword will find the website with or without the help of PPC ads. Exactly how many of these people will reach the website and make a purchase is unknown.

again this is all based on a no competition scenario


If ALL of the people who click on the ad would have made a purchase on the website anyway, the ROI from the advertising expense would be Zero.

Again denies competition.

The % of people who would NOT make a purchase because the ad is NOT there to click on, is the true measure of the ROI. Because the behavior of people can not be tracked or measured without serving them an ad, the ROI can not be tracked or measured. (No control group)

If we don’t know if it’s working or not, than why are we spending millions of dollars on it?
Why not spend the money on something else or just save it?

Are you starting to see the blind spot?

XXXXXXXXXXXXXXXXXXXX

We also do quite a bit of Banner advertising but this is not perceived as smoke and mirrors because the ROI reporting is much more accurate and accountable than search due to the ability to measure test and control groups against each other for the same time period.

This can be done for search also.... run multiple ads with different landing pages.



We can run a test where we show half the audience a real banner and the other half the audience a bogus banner for a charitable organization or whatever.

Once you have been segmented into one group or the other you will continue to stay in that group. You will either see real banner ads everywhere you look or the bogus banners for the duration of the test or until you delete your cookies.

Several, Many, Most or All of the people who are only served the bogus banners will make a purchase anyway.

Let’s say this is the results…
7,000 people who only saw or clicked the bogus banners made a purchase anyway.
10,000 people who saw the real banners made a purchase.

The real banners in this example show a 3,000 lift in orders.
The ROI contribution of the banners would be attributed to 3,000 orders. (Not 10,000)

XXXXXXXXXXXXXXXXXXXX

This sort of test and control methodology is not available with search.
This is why the ROI can not be accurately measured.

Again when your measurements start out as faulty then you can make any claims. Search can duplicate this... you can have someone cookied and watch future movements etc... claiming something for areas independent of search does not come with implicit correctness or accuracy





Before you can accurately determine ROI from search you would need the ability to track people who search for a keyword or group of keywords and are NOT served a PPC ad. This will tell you how many of those people will make the purchase anyway or more importantly how many will not.

Google and the other search engines would need to allow advertisers to purchase a tracking pixel instead of the actual PPC ad for a % of the tested keyword queries. People in one group or the other would need to stay in that group and either see the PPC ads or not for the duration of the test or until they delete their cookies.

Until this happens, we have no way to determine if the millions of dollars spent on PPC is a wise decision or not. If we don’t know of there is any ROI or not, then why are we spending the money?

You can get the information on how many searches are performed for any keyword and then you can see if the person comes through organic if you have solid tracking and analytics - you can also cookie a user and follow other behavior.... you should not narrow the discussion by taking an aggressive stance. People must stay in one group or another?

[Reality Check]

Wow!

A response from the guy who wrote the book on the subject! I am ordering it today! I sincerely appreciate this level of help.

Point #1
“Someone of good faith in the organization needs to believe that there is incremental revenue that can be created through a paid search program.”

“When some methodology is set up to attempt to prove that, they need to assess the information fairly and open-mindedly.”

To paraphrase Yoda… It is, or it is not… There is no “Belief”.
The smart folks in finance and accounting aren’t familiar with this stuff. The team who is in charge of the traditional media is familiar enough to be dangerous and they have more influence than I do. This is not a faith based expense. It’s strictly a business decision. If it’s working, we’ll do more of it, if it’s not working we’ll significantly cut back on it. People with more influence than me say it’s very likely that it’s not working.

Point #2
“You're fighting a rhetorical battle against someone who won't let you win the argument.”

This is how I view the situation as well. I can incontrovertibly prove the banner advertising works but not search.

Point #3
If I stop all promotion, guess what: no growth, fewer and fewer sales.

Not necessarily… With all the money we spend on PPC we could probably purchase a Blimp. (Of course we would never do that) But… It could be argued that the message on the Blinp would be more cost effective at driving traffic and sales on the website than PPC because the ROI for PPC can’t be proven any better than messages on the Blimp.

Point #4
“If the campaign is large enough, you could turn it on for a month, and then off for a month, and look at total revenues. Did you move the needle?

Nope… “Sales volume and visits can very greatly due to competitive conditions and seasonality. The typical % variance is grater than the usual contribution from SEM”

We have a large budget but if we shut it all down for a month and revenue dropped by a Million dollars, this could easily be attributed to competitive conditions and seasonality. It wouldn’t make a large enough dent to raise any flags. Revenue might even go up during this time depending on what season it is. It wouldn’t be a statistically significant test.

Point #5
“This argument could be made in connection with any form of advertising, though. Are the other methods being measured? Why would someone want proof that one method works, and not demand equal proof of the other methods' incremental value?”

Nope… Per the example above I can perform much better tests with Banner ads than search. They want the same proof that I can deliver for other forms of advertisement. If I can’t provide this proof than why not buy the Blimp instead?

Point #6
“Some of the argument behind this may come from math proposed by analysts like Jim Novo, trying to help companies avoid "cannibalizing" their channels by overspending in PPC where they don't have to (because, let's say, they appear in organic etc.). Not all of Jim's premises hold with regard to today's search landscape, IMHO. It's haggling over nickels and dimes, usually with regard to very large companies who are quite wasteful in other areas of their advertising.”

”You're in a political type argument, it sounds like. Send them my book.”

The people here are smart and only want what’s best for the company. People in the finance and accounting departments don’t understand search as well as they do math and facts. These nickels and dimes quickly ad up to millions of dollars. The other problem is the decision makers were sold a story that implied search was more accountable and the ROI generated could be measured. Now they aren’t sure so this brings up the question… “Why are we doing it? We aren’t doing it for “Branding”… That’s much better done with television.

I disagree with the “political” argument somewhat… These people are smart who are trying to make the best decision they can for the grater good of the company. If they understand one advertising medium better than the other, than that’s the one that will likely win. Almost all offline advertising promotions are designed to drive people to the website. This is easy to understand.

Point #7
“Is there a broad-brush assumption that the company is getting a whole lot of juice from organic rankings. True or untrue?”

Not true… We don’t have very good organic rankings. (I’m working on it)
We do have several dozen online distributors all promoting us along with our competitors. I can talk all day about vertical creep, one box results, etc, but people’s eyes seem to start glazing over after a few seconds.

We have good analytics, but the signal to noise ratio is too great dismiss an argument that if we turn off one distribution channel the others will pick up the slack. If PPC was our only source of traffic and distribution we could do this easily but that’s not the case.

Point #8
“There is only so much you can do to sell something to someone who isn't buying.”

I agree… However I can show incontrovertible facts and math that shows a very cost effective and statically significant lift in revenue with banner advertising. It’s not a subjective decision. It works and I can prove it. Not using banner advertising would be an irresponsible business decision. On the other hand… The ROI from SEM can not be measured accurately so there is no business case for this decision. It becomes subjective. Do I like sexy Blimps or PPC text links?

Point #9
At the very least do they not understand the market research value of 2-3 months of tracked paid search campaigns on a wide variety of keywords? Wouldn't they learn more about which landing pages convert if they ran systematic campaigns? Geez.

Nope… As you pointed out earlier… I’m fighting a rhetorical battle against someone who won't let me win the argument. We can have all the landing pages we want and I can do A/B testing to make the copy of the PPC ad better to improve click thru rates and tweak the landing pages all I want but if these people would have purchased anyway or if a Blimp can do the same job cheaper than what’s the point?

[AussieWebmaster]

Don't think you can achieve anything purer on the banner side... PPC can be measured as tightly if not more so.

[Reality Check]

Thanks Barry,

Yes! That’s an excellent example of my problem!
"Technically”, the cookie sees the Click-Visit-Conversion due to the brand name keyword. Our analytics package and ad agency both report this as a Positive result. The $1.00 click created a $100.00 order.

“Technically” the cost to revenue ratio % and ROI for brand name PPC queries were excellent. However… This reporting is easily confused with "Reality"

This is what I call a “False Positive" result.

The ads may or may not be the driving factor in revenue that is reported, or people might click on ads only because they are there but they may or may not be responsible for producing the sales that are "Technically" attributed to the keyword at all.

Search engines like Google and Ad agencies get paid to sell advertising, not raise questions about the reporting limitations. The decision makers over here realize this and are skeptical of the ROI reports as they should be.

For example:
We were paying a small fortune for our brand name as a keyword which we ranked in the #1 spot for. About 1/3 of total visits from brand name queries were generated from PPC ads.

I analyzed our analytics data and determined that the % of brand name visits was a surprisingly consistent ratio week over week when compared to overall traffic generated by each major search engine. Traffic goes up and down significantly but the brand to generic ratio stays about the same.

Hypothesis:
If we shut off the PPC traffic for that keyword, our traffic ratio for that keyword should drop by 1/3

Result:
Nothing happened. The ratio did not drop by 1/3. It did not drop at all. It stayed exactly the same. We turned the keyword back on and still nothing. So we turned it back off for good and checked it again and still nothing.

Conclusion:
The positive ROI data for the keyword buy was a “False Positive Result”. We would have received the traffic and orders anyway. There was no business justification for the expense as measured by a direct response metric. The expense was eliminated because it may have had a “Branding” effect but the cost for this effect can be produced more cost effectively with television.

The methodology for test above works for our brand name but I wouldn’t recommend it for generic queries. Even if the traffic ratio for the generic keyword drops to zero, there is no way to prove that they would or would not be able to find us and make a purchase through the natural comparison shopping process.

Determining “Actual ROI” with search is almost impossible because there is no way to determine lift versus a control group who didn’t see the PPC ads during the same time period for the same keyword or list of keywords.
This test above saved us a boat load of cash but it also casts a shadow over the rest of my SEM efforts that I truly believe deliver an excellent value. It creates a difficult situation for me that I’m struggling to reconcile.

[Reality Check]

Thanks AussieWebmaster!

Point #1
“SEM is the most accurately measured online marketing method.”

Not exactly…
As demonstrated above…

With banner advertising, we can run a test where we show half the audience a real banner and the other half the audience a bogus banner for a charitable organization or whatever...

This sort of transparency of ad effectiveness is not likely to happen any time soon because many advertisers would suddenly realize the ROI they had assumed to be correct, is much less than previously thought. Others might realize it was better than they thought but allowing this sort of transparency could be risky.

Point #2
“Any caveats - and blind tests - can just as easily be used as a counter for any marketing efforts.”

I’m not sure what blind tests you are referring to. As far as I know… The ads are either there or not. You can’t segment the audience into two groups that either continue to see the ads or not for a period of time.

Point #3
“Any advanced web analytics program can give you the results for paid and organic conversions.”

“Technically” Yes… But not necessarily as demonstrated in the response to Barry.

Point #4
Okay and if you don't pay your banner bill they don't appear - or if you do not buy all possible impressions your banner is rotated - this is an issue of campaign budget limits.

We always pay the banner bill because we can prove they work. Basically, we can pay to show a bogus banner to half the audience and measure the difference in response from the two groups. You can’t do this with search.

Point #5
If you do not have an organic listing on the page.... you can pretty much measure the fact that all searchers did not go to your website from the search page.....

Yes… It is reasonable to assume that these people would have just as likely reached our website via some other method than search. They can directly enter the URL that they saw on the TV commercial that was financed from the money we saved from not participating in PPC.

Point #6
Me: “You can NOT track and measure the behavior of these people if there is no ad to click on.”
AussieWebmaster: Say the same thing in reverse always adds momentum against a proposition.

Not true because you CAN measure the behavior of people who do NOT see our banner ads. People who do not see our banner ads are much less likely to not make a purchase. We know this for a fact because we can measure both groups.

Point #7
“Sounds like the tree falling in the forrest routine....
The first hint to this is if they are about to buy and you have competitors who are advertising - you are going to lose potential customers - this is not a vacuum - there is no simple answer but you should consider the competition in there - if you have none then why bother advertising at all.”

“You are going to lose potential customers”
This is true… But exactly how many customers are we talking about? Is it one customer or all of them? This number can not be accurately determined without measuring the behavior of people not exposed to the ads.

If the number is small, it doesn’t justify spending millions of dollars to keep them. It might be more economical to save the money.

There is a lot of competition but it is unknown how many of the people who are not exposed to the ads will purchase from them.

We can run dozens of landing pages and do all the A/B testing we want but the ROI from search can not be effectively measured against a control group. If a TV commercial may or may not be more effective at persuading the customer to make a purchase then why not sacrifice PPC to give it a try?

Point #8
“Search can duplicate this... [Test and control situation available with other online media] you can have someone cookied and watch future movements etc... claiming something for areas independent of search does not come with implicit correctness or accuracy.”

I don’t believe this is true. I am not aware of a way that I can cookie a person searching for “Blue Widgets” unless I have an ad displayed and they click on it. If I could cookie them with out a PPC ad for them to click on, I could track them and determine how likely they would be to purchase our product without the help of an ad. Unfortunately, I can’t do this.

Point #9
“You can get the information on how many searches are performed for any keyword and then you can see if the person comes through organic if you have solid tracking and analytics - you can also cookie a user and follow other behavior.... you should not narrow the discussion by taking an aggressive stance. People must stay in one group or another?”

Again… We can not cookie them if there is no ad to click on. We have solid tracking but we can’t begin tracking them before they click the PPC ad.

I’m just playing Devils Advocate. I completely agree with you on most points but I'm having a hard time winning that argument.

[recoveringtechie]

It seems what you are arguing is how much impact your entire marketing mix is influencing your overall sales. Since you have a strong brand it's always a tough question because everything you do typically increases your brand awareness. Some would infact argue that measuring branding is futile.

I can certainly see holes in the banner measurement of impressions you lay out given the amount of cookie deletion and it's impact over large time periods needed to accurately measure brand awareness. This would make me question how certain you really were about the effectiveness of the banner campaigns. I would also wonder whether you were measuring ROI with fully loaded costs or not.

My take on this is that measurement tools can give you a good idea of relative performance of your activities but one should never go sofar as to think of them as literal measurments because no matter how you look at marketing activities (online or offline) there's always things that impact your ability to get to perfect accounting numbers.

As for whether search marketing is working for you I would then look at it in two ways, (a) does the behavior suggest an increased amount of impulse buying or a general shortening of your sales cycle from click to purchase? (b) does the existence of the keyword program increase your brand awareness in other ways which are similar (and perhaps cheaper) than other brand marketing?

I would suggest that all things being equal a click on a banner might take longer to turn into a sale than a click on a keyword if the keyword is specific enough. This has a lot to do with how people progress through their sales cycle and their proximity to the sale.

[Reality Check]

Thanks recoveringtechie,

I agree with every point. Unfortunately… The SEM campaign is measured by a strict direct response metric for cost of advertising versus revenue generated. Branding and raising awareness is not a goal or even a consideration.

I agree… Measuring branding is futile. If SEM is only good for immeasurable “Branding” purposes, I’m out of a job, so that’s the last argument I’ll be trying to make.

We are only concerned about the ROI of the click. Does it economically generate revenue or are people just as likely to make the same purchase, with or without the help of the PPC ad?

The banner test isn’t perfect, but it’s fine with me if some people are deleting their cookies because this will have a negative impact on the test results. If anything, we are underestimating the impact of the campaign. The more people delete their cookies the les conversions the banners can take credit for. I can show the positive numbers and say with confidence that “These numbers are probably even better than this because of cookie deletion.”

With search… Someone can say the numbers are likely much worse than this because a large % are just as likely to visit the website anyway. I can’t prove this to be wrong.

Even with banner advertising, we don’t care about the branding impact. If the placements per publisher don’t produce sales, we quickly cut that publisher from the banner campaign.

No matter what… The cost of the advertising placement can not exceed the profit margin on the sale or the ROI will be a negative number.

If the banner ads or SEM have a branding impact… Bonus! But… We only care about achieving a positive ROI per ad placement regardless of the online advertising medium.

We don’t expect perfect ROI numbers. I just need to show ROI numbers that can not be easily dismissed as being purely smoke and mirrors.

Quote:
“(a) does the behavior suggest an increased amount of impulse buying or a general shortening of your sales cycle from click to purchase? (b) does the existence of the keyword program increase your brand awareness in other ways which are similar (and perhaps cheaper) than other brand marketing?”

I wish those were the questions but they are not. The question is; are people just as likely to purchase with or without the PPC advertising expense. If so… By how much?

Will Some, Many, Most or All of the people still make a purchase anyway?
Without tracking a control group, I can’t answer that question in a way that will satisfy the smart folks in the finance and accounting departments.

Quote:
“I would suggest that all things being equal a click on a banner might take longer to turn into a sale than a click on a keyword if the keyword is specific enough. This has a lot to do with how people progress through their sales cycle and their proximity to the sale.”

I agree… However, if people who see the banner are just as likely to make a purchase with or without the PPC keyword, than why are we spending millions of dollars on it?

The questioning of the SEM campaign usually follows these lines…

• Show me the money! Does it work or not?
• OK then… Prove it!
• If you can’t prove that it works, then it probably doesn’t work.

It’s a bit like debating with someone who thinks the earth is flat but I don’t have the proper tools to show otherwise.

[Reality Check]

To help the conversation, I will attempt to show how the Finance and Accounting departments might see the situation.

Let’s say our product sells on our website for $100.00
This price would give us a 20% profit margin or $20.00 per unit.

The average cost per click is $1.00
Our conversion ratio is 10%
Or $10.00 PPC Cost & $10.00 Profit per order.

So in this situation…
Our ad agency and analytics package would both say…
1 Million dollars in PPC sold 100,000 units and we make 1 Million in ROI.

This would be great! The folks in Finance and Accounting would all say that with a profit margin of 10%!!! - PPC is our most profitable and efficient distribution channel. (Great ROI)

The problem is…
It’s a reasonable assumption to say that a significant % of those 1 Million visitors would have visited the website anyway and made a purchase with or without the help of PPC.

If that is the case… The % profit is inversely proportional to the % of people who would have made the purchase with or without the help of PPC.

If 25% of the people would make the purchase anyway, than this would drops the ROI per unit down to $5.00 and PPC is at best a mediocre distribution channel and certainly not our best option.

If 50% of our PPC traffic would have made the purchase with or without PPC then that leaves us with Zero ROI.

If more than 50% of the visitors would have made the purchase anyway, we are loosing money on every order. If all the people who clicked our ads would have made the purchase anyway then we just lost 1 Million dollars!

It could be argued that turning a 1 Million dollar deficit into a 3 Million dollar profit is as easy as killing the SEM campaigns.

This sounds like crazy talk to me but it’s a simple argument and I can’t figure out how to counter this argument.

If the major search engines would allow us to pay to place a "tracking pixel" in the SERPS for a set of keywords or the entire portfolio of keywords instead of the PPC ad, we would be able to accurately determine the lift in orders directly responsible for the ad placements and ROI of the expense.

In my opinion, this is a huge missing piece of the SEM puzzle.
I refuse to believe that the smart folks at the search engines can’t do it.

Until then, I believe more and more large advertisers will have their PPC budgets cut because we can’t prove if PPC working or not

[mcanerin]

I think you may have already outlined a potential solution to your problem, Reality Check.

I’m not sure what blind tests you are referring to. As far as I know… The ads are either there or not. You can’t segment the audience into two groups that either continue to see the ads or not for a period of time.

I think you can. Do the same thing you do with banners:

We always pay the banner bill because we can prove they work. Basically, we can pay to show a bogus banner to half the audience and measure the difference in response from the two groups. You can’t do this with search.

Sure you can. Simply bid with PPC using same/similar ads for a different site or subsite. Heck, for a spend that size you could alternate every hour (or any other way you wanted to) for as long as it takes to get the information necessary, as long as you were using PPC software capable of making those changes on the fly. Use a second company if necessary.

Treat the PPC as you would the banner ads, just trickier to control. It sounds like all you really need to do is show the ROI (or lack thereof).

I don't doubt that there are circumstances where PPC may not help, but I've personally not run into any, unless they were very poorly optimized. There have been times where I've seen PPC sending a company rocketing to the poorhouse because they were bidding stupidly, but that's the only time. I'll allow that there may be exceptions to anything, however.

But I'd like to see the data (much like the bean counters at your company).

I think that if you put some thought and planning into it, you could use PPC to emulate the effect and testing that you see with banner ads. Since they accept the banner ad ROI measurement, that is the standard you would/should be held to, rather than some theoretical perfect scenario. I would focus on that, since it's measurable, realistic and (I believe) attainable.

Just thinking out loud,

Ian

[Reality Check]

Thanks Ian!

Quote:
Simply bid with PPC using same/similar ads for a different site or subsite.

Hmmm... I’ll need to check in to this. It might work but I’ll need to research the possibilities before I can say if it will work or not in my situation.

The website would need to be relevant enough to qualify for the ad group but it shouldn’t be a competing website. It would need to be compelling enough to click on so we can cookie as many people as possible but not specifically promote our product category.

I think this might work! I’ll need to think about it for the rest of the weekend but it sounds promising.

I apologize to anyone who posted something along the same line as this earlier but somehow the concept didn’t register until now.

Thanks again. I’ll let you all know what the situation is as soon as I have some news.

[mcanerin]

I'd be interested in what the result is, Reality Check - regardless of whether it's what we are hoping for or not.

All data (even negative) is good data, and helps everyone do their job a little better.

Cheers, and good hunting! ;)

Ian

[Fishcharge]

Hi there. I'm new to all this, so I apologise in advance if I'm missing some glaringly obvious thing in my comments, but you've got to learn sometime I suppose.

Having read through the previous posts, it seems to me that the problem could be approached from a different angle. Here is what I mean. Reality Check states that:


The problem is…
It’s a reasonable assumption to say that a significant % of those 1 Million visitors would have visited the website anyway and made a purchase with or without the help of PPC.

I'm not sure I agree that this is a reasonable assumption and this seems to come to a decision at the strategic level, not the tactical one. Why are PPC campaigns being run in areas where you're making assumptions that a significant % of traffic will arrive at your site and purchase anyway? If in reality a significant % of traffic convert all the way through to purchase - this should be reflected in your traffic analysis. Given the option between clicking on an organic link vs. a PPC ad, I think most consumers will choose the organic link. Which brings me back to the original question - why are PPC campaings being run in areas where most of your sales are being generated by organic search results?

In my mind PPC campaigns should be used to close the gaps that your SEO strategy failed to.

Secondly, it also seems to me that your analysis stops a little early. Wouldn't it help to look at the types of customer that your various strategies are creating? Which search phrases (both organic and PPC) acquire the most valuable customers (by whichever indicators you use to measure value) or which search phrases generate customers who make second and third purchases, and where did they come from (organic/PPC)? Thus you're moving away from the idea that you need to focussing on PPC ROI to acquiring profitable customers with positive ROI for your long-term business. While click ROI is a useful measure, I don't think it is necessarily the most valuable one.

Thanks,

Fishcharge.

[Mikkel deMib Svendsen]

I don't think the problem is the answers you get but the questions you ask! You take a broad spectum media like search and want to meassure it with a narrow set of metrics specified for other medias. No matter how you do this you will fail. You will not come up with the answers you seek because the questions are wrong.

If you set up the "right" metrics you can prove that a car can't possible drive, flies can't fly, and TV-advertising dosn't work at all. Its all up to the questions you ask, the metrics you use and the methology you pick. But even if you prove cars can't drive my car still takes me to work every day and the flies bug me at night :)

If you want to meassure search marketing only by a simple set of ROI metrics set for banners you won't get what you are looking for. It's a hopeless task.

My best advise for a client like that would be: Drop search marketing and leave the market to your competitors :)

[shor]

Interesting thread, great responses. I've been in a similar situation to RealityCheck. I work at a big corporate, millions in marketing, direct response metrics as KPIs etc. Based on my experiences I have a few recommendations.

If your company can afford to spend $millions on marketing, they could always divert money from their next brand awareness survey into a focus group:
"If I ask the research group to look for a product, how many end up finding my company's red widget unaided by PPC?". (Watch all those users click onto your competitor's PPC ads!) Wouldn't that give you measurable and accurate data that answers the question? Expensive way to do it, but it is viable...

Or subscribe to HitWise and check the share of clickstream data between your company and competitor websites when you have PPC on and off. Do you see that trend of increased search engine traffic referrals to your competitor when you have PPC off? That's the company that is taking market share from you. Those users will NOT find their own way to your website because if your competition is anyway decent, they are well on the way to converting those users into customers.

I understand that SEM's contribution to your total revenue is less than market variance but you can still create tests that answer the question. You say have several dozen distribution channels, well choose a single product and market that product through a limited and measurable set of distribution channels that includes SEM. Eg. Red Widget is only advertised through banner ads and ppc ads. I take away banner ads, what revenue do I get from PPC? I take away banner ads, what revenue do I get? I then take off both channels and I am left with revenue contributed by direct typed/bookmarked visitors.

Finally, get a SEM champion! This is the upper-management dude or dudette that tells the head of Finance to get off your back. Sell your story and argument to them, and it'll be a lot easier to keep those accounting rodents away from your cheese :)

If you want to meassure search marketing only by a simple set of ROI metrics set for banners you won't get what you are looking for. It's a hopeless task.

My best advise for a client like that would be: Drop search marketing and leave the market to your competitors :)
Otherwise Mikkel is correct - please leave more market share for the rest of us! ;)

[mugshot]

The problem is…
It’s a reasonable assumption to say that a significant % of those 1 Million visitors would have visited the website anyway and made a purchase with or without the help of PPC.

Assumptions are never reasonable :) We can then also assume that if you drop SEM, then the profit % will also decrease? Or we can assume that if those purchasing from banner ads will purchase anyways without the banner ads?

I guess in short, assumptions will only create more assumptions - at best.

[ShoeMoney]

I do not understand why people make something so simple so complex.

You spend X you make X.. what is the issue?

[MaKa]

Not sure whether it will work, but something I tought of:

You say due too seasonal fluctioations you can't do an A/B test over a longer time span. You said the short hourly A/B test suggested earlier might work.

Could you get around your problems by using regional targeting? Run a campaign for a month in half the regions your targeting then switch the regions around for the next month and use the data from the non campaign running regions as a benchmark.

[Reality Check]

Thanks ShoeMoney,

Let’s say you have massive name recognition and popularity.

You could ask the average guy on the street to say the first word that pops into his head when you say the word “Ringtones” and his response is almost always “ShoeMoney”.

Here are the questions you would need to ask yourself…

If the keyword “ShoeMoney” cost you 1 Million dollars per year because everyone is trying to ride your wave, would you still spend that kind of cash if you were in the #1 spot organically anyway?

Your analytics package says that Paid clicks on keyword “ShoeMoney” have an excellent conversion rate (Duh) and the PPC keyword is responsible for driving a “Huge” portion of your ROI.

Do you honestly believe that removing that 1 Million dollar expense would cause a “Huge” loss of revenue? Or would you think along the lines of it might be worth keeping for vanity reasons only?

If “Ringtones = ShoeMoney in the mind of the consumer…
With this kind of brand name recognition, you could assume someone shopping for “Ringtones” will hit your site at some point in the comparison shopping process anyway because you are the obvious market leader.

Would you still buy the keyword “Ringtones” if it cost you 1 Million dollars per year?
What % of the people searching for Ringtones will hit your website at some point regardless of your PPC activities?

If your profit margin on every Ringtone sold is super thin and every click is eating away at your profit, you might seriously think about trimming the keyword list.

If you were working with an ad agency… They would say that those keywords drive a “Huge” % of your total revenue. “You spend X you make X”.
Removing the keywords from your portfolio could cause a “Huge” loss of revenue.

Do you believe this? Or do you believe that almost all of them will buy anyway? Or at least more than enough to compensate for the 1 Million dollars you are currently paying out to Google for PPC ads.

What is the “Real” return on your multimillion dollar investment?
What is the best business decision to maximize your profits?
(This is real cash that you could keep for yourself)

The simple solution could be to cut some of the most expensive keywords that your analytics package says is driving most of your ROI.

The complex solution could be to explain to your investors who don’t understand search, why you are paying Google 2 Million dollars for something that may or may not actualy be needed.

[Reality Check]

Thanks Maka,

I was actually thinking of driving all of our PPC traffic to a neutral website for the same keyword list with a relevant and compelling ad that people would click on. Hopefully this would give me enough cookie data to analyze what % of the traffic will purchase anyway.

Unlike shutting it all off for a while… This would allow us to measure the behavior of people who didn’t see the ad.

A regional target test like you described could work but at this time I believe that diverting the traffic might produce better data on visitor behavior.

[Reality Check]

Thanks Fishcharge,

The assumption that everyone will find the website no matter what is something that I don’t agree with it either. But… I do agree that at least a few of those people will find the website.

We don’t have great organic rankings for most of our PPC list. We rank in the #1 spot for our brand name but not so well for generic product queries. We are currently using PPC to fill the gaps as you suggested.

We don’t differentiate the value of the customer beyond how much they spend on a purchase. Not for online advertising anyway. It’s a simple direct response campaign. The cost of advertising can’t exceed the profit margin for this sale, regardless of some other long term value the customer might have. Either we made a profit on the sale influenced by PPC or we didn’t.

[Reality Check]

Thanks Mikkel deMib Svendsen,

I agree that the questions are more of a problem than our PPC campaign. Unfortunately, they aren’t stupid questions and I need to answer them. It’s my job and I like my job

[Reality Check]

Thanks again Mugshot,

I agree that the assumptions are unreasonable. The assumption that if we drop SEM then the profit % will also decrease is a battle I’m loosing right now.

It’s easy for others to pose questions and state assumptions implying that the ROI from SEM can not be measured accurately but it’s difficult for me to prove this logic to be false.

[nutella]

If the keyword “ShoeMoney” cost you 1 Million dollars per year because everyone is trying to ride your wave, would you still spend that kind of cash if you were in the #1 spot organically anyway?

We don’t have great organic rankings for most of our PPC list. We rank in the #1 spot for our brand name but not so well for generic product queries. We are currently using PPC to fill the gaps as you suggested.

That's strange. If you do not even have top ranking for most of the PPC terms you target, couldn't you simply argue, in terms of online users, you simply do not get "any" sales from search users w/out PPC.

Going back to the "shoemoney" ring-tone example, sure, I'd love to buy "shoemoney" but am I desperate enough for "shoemoney" that if I don't find it online in 10 seconds, I'll continue to search for it? Or better yet, will I care if , in 10 seconds, I find "shoemoney" online from Verizon instead of TMobile?

Visibility of your brand does not necessarily mean visibility of your site/online presence.

[Reality Check]

Thanks Nutella,

All true. However… I can not make any arguments. I need to show incontrovertible proof.

[mcanerin]

BTW, this is an excellent thread, and an important one.

I just wanted to thank everyone (especially Reality Check) for discussing it. It's not often threads about the business of SEO show up - all too often it's about technical issues and rumors rather than ROI and spend analytics.

The fact that this might be an uncomfortable question/issue for some people means it's an important one that should be asked and addressed. :)

For one thing, I'm pretty sure someone will ask me it at some point (I have a lot of large clients), so I'd rather work out the answers and data here, among friends and colleagues, than on the spot in a boardroom after an AGM, if you catch my meaning...

Ian

[Fishcharge]

Reality Check,

How about running an A/B/C test like this:

A=#1 rank PPC
B= #4 rank PPC
C=No PPC (control)

Monday = A
Tuesday = B
Wednesday = C
Thursday = A
Friday = B

You get the idea. You run this test for 3 weeks, so that by the end of the period A,B & C will have run on every day and you can at least compare those results.

Wish I could take credit for the method, but found it here (http://www.jimnovo.com/newsletter-2-2006.htm#Q1)

Agree with the others that this is an excellent discussion.

[Reality Check]

Thanks for the positive feedback Ian!

It’s an uncomfortable subject for me as well. It sucks to have the value of your work questioned.

On the other hand… I can sympathize with the bean counters as well. This is not a passion or labor of love for the folks in Finance. It’s just business.

Everyone around here is friendly so it’s not a childish game of “Gotcha!” Everyone genuinely wants what’s best for the future of the company.

No one wants an Enron type situation where revenue is attributed to the wrong sources. If I can prove its working, we’ll probably do more of it and if I can’t… then we’ll likely reallocate most of those funds to something that can be proven. It’s strictly a business decision based on verifiable facts and evidence.

(IMHO) If you work for a large corporation or have one as a client, it’s only a matter of time before you are asked some hard questions about the revenue and ROI attributed to PPC search.

I’m not in the hot seat right now but I can see which way things are going. I believe just about any SEM professional would think we have a sophisticated and well run campaign set up but the standards for performance are getting tougher as I guess it should.

I’ll share as much as I can as I go through this process. It might take a while before I can share results because whatever test methodology I use will need to be approved etc, and a test can take a month or longer to conduct and analyze.

[Reality Check]

Thanks again Fishcharge,

Thanks for the link! I remember Andrew Goodman mentioned Jim Novo in his post but I hadn’t gotten around to searching for it. It seems like Jim has a background in Finance or at least has a good sense for what makes those people tick. I don’t think that method will work in my situation but I need to think about it a bit longer before I can rule it out.

When I first posted this thread I was frustrated and thought this was a problem that can’t be solved but I’m starting to believe there may be more than one way to get results that will satisfy the critics.

[Reality Check]

Thanks for the positive feedback Shor. I apologize that I accidentally skipped over your reply.

A focus group might work. We could kill the PPC campaign long enough to run a focus group and document their behavior with and without our ads.

We checked into HitWise and asked them for a few samples of traffic data that corresponds to spikes in traffic and sales due to promotions and special events, etc. The results from Hitwise wasn’t any better than Alexia traffic results in my opinion. I haven’t tried Trellian but Dave and his crew over there seem like they have a product worth trying.

I had a an SEM champion… Now he’s not so sure. As I keep repeating, the arguments are too simple and easy to understand and I don't have any easy answers yet.

Mikkel is correct! But that’s another problem.
We could give up and let our SEM campaign be run by a thousand random SEO and arbitrage affiliates on a pay for performance basis but I believe that we would benefit more in the long run if we were able to maintain control of the situation.

You said you were in a similar situation. What was the result? Were you able to verify the value through a focus group or did the heat just die down by it’s self?

[shor]

You said you were in a similar situation. What was the result? Were you able to verify the value through a focus group or did the heat just die down by it’s self?

Our SEM campaign is a large enough % of total distribution to easily verify the change when PPC is turned off. Net KPIs fell drastically without PPC (indicating that our competitors in natural/paid had taken over market share). We also had fairly extensive historical data to data mine and our markets have a well known seasonal trend. This also let our SEM champion walk in with clear idea of how SEM performs versus other channels. Understandly much easier to solve than your predicament :P

I've seen focus groups used for similar reasons to your objective but honestly, that's an expensive way to go about it :P However, if that is what it takes to prove the naysayers wrong, there's nothing stronger than seeing Jo Blo, your average surfer told that he/she is to buy a red widget, turn to a competitor's website through a PPC ad. Or when you have no natural listings and you lose your PPC listing, Jo Blo has no where else to go but a competitor.

[Reality Check]

Thanks again Shor,

Like most companies, we have two main kinds of competitors. We have direct competitors and resellers who sell our product as well as our direct competitors products.

I can’t visualize any other situation than the consumer visiting our competitor’s websites if we remove the ads. The only question is how many people will go to a competitor and then comparison shop them against our official website to see if they can find a better price or because they feel more comfortable making a purchase directly from the source versus a reseller.

The best case scenario is that our brand is strong enough to dislodge enough consumers from the competitions websites to compensate for the cost of PPC.

It sounds like a long shot to me, but I still need to prove that this is either happening or not and to what extent.

[Robert_Charlton]

The best case scenario is that our brand is strong enough to dislodge enough consumers from the competitions websites to compensate for the cost of PPC.

You might want to consider, though, how much your PPC presence or lack thereof might affect your branding.

Greg Jarboe at SES has also pointed to a symbiotic effect between top Adwords and top organic listings, which increases the click-through rates on both. The reference I have at hand for this is the following SearchDay article, which might be useful to you...

Balancing Paid and Organic Search Listings
http://searchenginewatch.com/searchday/article.php/3095871

Content optimization for organic results and pay-per-click advertising provide complementary benefits, the panelists agreed. Jarboe described a study at SEO-PR that revealed click-through rates tripled for a site positioned in top spots for both natural and paid listings.

[Reality Check]

Thanks Robert,

I agree.

The best situation is to create what I call a “Search Engine Blockade” where you occupy as much of the real estate in the SERPS as possible including paid, natural and one box results.

The problem with the SEO-PR study you pointed out is…
“Click-through rates tripled” (Who cares?) Our finance department couldn’t care less about “Click-Through Rates”. It’s completely irrelevant. Visits do not necessarily equal profit. We only care about profit.

These are questions I would likely be asked in a similar situation. (Show me the money)

How much did the extra clicks or visits cost?
Did those extra visits translate into profitable sales?
What was the PPC cost to revenue ratio %?
Was this ratio within our profit margin?

I was fortunate enough to be at SES in San Jose last year and I believe I saw that presentation but I don’t remember any details on how the extra traffic was attributed to a profitable outcome. It is implied that the extra visits were a good thing, but they could have just as easily lost money on the deal. Who knows? Maybe profit wasn’t the goal and that’s ok.

Our finance department doesn’t care at all about “Branding”. The cost of the clicks either led to a profitable transaction or the PPC cost exceeded our profit margin and caused a loss. Anything that is a possible source of loss is very likely to be cut.

The profit generated by PPC needs to be documented in a way that is incontrovertible. The smart folks in Finance aren’t going to stick their necks out and go on the record as saying our PPC campaigns are profitable unless they see incontrovertible facts that prove this claim.

I am finding that proving ROI claims for PPC can be a serious challenge.

[Reality Check]

360i and SearchIgnite just released a whitepaper where the study was to track the value of the entire click path or the keyword stacking effect of the consumer. It’s worth reading and has a lot of great information.

http://searchenginewatch.com/searchday/article.php/3608456

However… (IMO) The most interesting quote in the article is this:

"We're also aware of other questions that have yet to be answered. How do interactions with natural search and other forms of interactive marketing affect the results? How do these findings impact transactional metrics such as average order size and return on ad spending? How does offline brand equity play a role?”

(IMO) Those are GREAT questions to ask. Especially for people who would like to prove the campaign is able to turn profit.

I’m sincerely looking forward to see how they answer those tough questions.

[farees]

I do not understand why people make something so simple so complex.

You spend X you make X.. what is the issue?

Well it’s not always as simple as that- well may be it is for smaller B2C segments. Talk to corporate Brand Managers and they will have a lot more to talk about Brand Marketing and PPC. The fact is- it’s always hard/tricky to measure Brand effectiveness and loyalty. How can you measure your brand effectiveness? How do you measure your ROI on Brand Marketing? I call it Return on Influence rather than the dreaded Investment :).

- Farees

[andrewgoodman]

Point #2
“Any caveats - and blind tests - can just as easily be used as a counter for any marketing efforts.”

I’m not sure what blind tests you are referring to. As far as I know… The ads are either there or not. You can’t segment the audience into two groups that either continue to see the ads or not for a period of time.



On this point, if you're looking to design an advanced experiment, sending half the paid search traffic from keyword group X to one landing page, and the other half to your dummy page, well, split-testing is built into the AdWords ad serving system, so it's possible.

However, it's not really feasible in the sense that these companies have editorial controls, both algorithmic and human, that will vet the quality/relevance of landing pages. That means they may not be treated equally, and your dummy page may well be rejected.

I suppose you can do this with banners, if sending half your traffic to a fake charitable organization is something the publishers of the websites you show up on are comfortable with, and if that's something the ad network permits. I'm not so sure that's fair to the user.

Can you run dummy ads in the New Yorker to test your marketing effectiveness? Not if they have anything to say about it.

Again, the bar seems like it is being set much higher for search, even though it is highly measurable with some constraints. Other media are typically more inexact and results are more often obfuscated.

Web analytics can be quite advanced. Look at the site usage profiles from users clicking through from various keyword groups, organic listings, banners, and other sources. It's not something I spend my days doing, but it's all out there. Perhaps the "cash savings on the company name" is coloring the whole debate. Paid search creates highly measurable results. Within normal limitations, of course.

How is someone ever to debate the "would have bought from us anyway" proposition? Would have? It's a counterfactual argument. I doubt it can be proven beyond a shadow of a doubt either way.

"Had there been no wheat boom in 1911, Canada's population would be half its current size today, and its GDP 60% of current levels. Discuss. Would the country have grown to current levels anyway?"

Who knows.

[shor]

360i and SearchIgnite just released a whitepaper where the study was to track the value of the entire click path or the keyword stacking effect of the consumer. It’s worth reading and has a lot of great information.

http://searchenginewatch.com/searchday/article.php/3608456

However… (IMO) The most interesting quote in the article is this:

"We're also aware of other questions that have yet to be answered. How do interactions with natural search and other forms of interactive marketing affect the results? How do these findings impact transactional metrics such as average order size and return on ad spending? How does offline brand equity play a role?”

(IMO) Those are GREAT questions to ask. Especially for people who would like to prove the campaign is able to turn profit.

I’m sincerely looking forward to see how they answer those tough questions.

I was going to quote that report as well, then realised it didn't answer your question at all! It does makes a great point on measuring the value of generic keywords early in the buying cycle. In my experience, attributing a single sale across multiple channels is something that few companies actually track and report on.

Re-emphasizing Andrew's point, it always puzzles me when other channels that use obscure or very broad performance indicators are given more leeway than SEM.

[Reality Check]

Thanks again Andrew!

It’s all in the planning stage and I’m still not 100% sure how I’m going to do it, but I was thinking about diverting 100% of our traffic to a neutral website that is not brand or product specific.

The editorial policy does make it tricky. I’m not sure if there is a perfect website to act as the recipient of the traffic. We may need to create one from scratch.

We don’t send people to fake charitable organizations. (We don’t need that kind of bad Karma) We pick a real one like the United Way or the Red Cross or whatever and they send us real banner ads of their choice to use. Everyone is happy with the whole process and a worthy organization receives a bunch of free advertising. (Good Karma)

I agree that the bar is set high with online advertising but it’s about the same as our email, direct mail and other “Direct Response” advertising vehicles.

I’m not an expert on direct mail but if you sent out 1 Million Dollars worth of brochures or offers in the mail and the response on the mailing didn’t produce a profit, you would probably stop doing that.

I guess it all depend on what your goal is. Pharmaceutical companies use PPC search for branding and it seems to work. Phizer can’t sell pills on their website, they purchase keywords like Migraine and the landing page provides information about the ailment and how their pill can help ease the symptoms. I think this is great for branding. The goal is to sell the researcher a lifetime supply of the expensive drugs.

Our campaign is different. Our SEM goal is to place the ad in front of the consumer when they are ready to make an expensive one-off purchase. We usually don’t get a second chance at the sale. The cost of the ads can not exceed the profit margin of the sales or we are operating at a loss.

The banners show what % would have bought from us anyway so we devalue the ROI on the total volume of banner ad conversions by that % and they still have a good ROI so we keep using them. Not using them would be a bad business decision.

With search, we don’t know what % would have bought from us anyway so it’s not a rhetorical question, it’s a real problem. (IMO) This missing number is necessary to evaluate the ROI.

(IMO) The search engines could help if they wanted to.

If I could pay to place a tracking Pixel in the SERPS instead of an ad, I could cookie everyone who searched for that keyword and determine the likelihood of them making a purchase on my website anyway.

Do you believe this feature; if it were available would help advertisers like me determine the effectiveness of their ad spending?

[projectphp]

If I could pay to place a tracking Pixel in the SERPS instead of an ad, I could cookie everyone who searched for that keyword and determine the likelihood of them making a purchase on my website anyway.
LMAO!!!! You have GOT TO BE KIDDING! Can you seriously imagine an SE allowing that? Just imagine how many pixel gifs every page would the page have if everyone wanted to do this, and how quickly.

I know you are frustrated, but man, that is a HUGE ask!

Do you believe this feature; if it were available would help advertisers like me determine the effectiveness of their ad spending?
Leaving aside the fact it is implausible, no, I don't. I don't really think the problem is that you need this exact data. The real problem is that you are expected to provide impossible amounts of info. And yes, impossible means what you think: impossible to get, with no amount of information ever likely to satisfy such an impossible ask :)

I have followed this, and I am left with one question: why can't you propose a crazy guestimate?

What I would say: "OK, so the ROI is hard to measure. Is anyone here making the claim that SEM leads to NO extra sales? (I assume the answer is no). Fine, then lets set a percentage of margin we need to hit with SEM that you will accept. If I can hit 70% of margin, will that be acceptable? What about 1%?" Then negotiate to a point everyone is happy with.

If there isn't a reduced margin, and people sincerely believe SEM leads to no extra sales, you have your real answer as to where this is headed (they are just trying to keep their own budget, are frightenned etc).

If not, then you have a figure to work towards, make damn sure you hit that figure, and the debate ends. From there, you can, over time and with a proven track record, lobby for an increase in the percentage of margin you are able to spend.

IMHO, your problem is political, not KPI based, and playing politics is the way to go. Build a history of hitting outrageously difficult targets, and then slowly petition for a more realistic ROI figure. Call it a game of massive bluff, but I would say this is the best way to get a strong foothold, and also to "out" the real motives behind why they are being so difficult (stop the new kid on the block taking my budgetitis, aka Donnie Wahlberg Syndrome).

[Reality Check]

Thanks Shor,

I agree… Generic keywords are certainly a key.

One strategy is to not be so keyword specific and evaluate the campaign as a whole. You can purchase research related generic keywords like “Red Widgets”, transactional keywords like “Cheap Red Widgets” and brand keywords like “Brand X Red Widgets” and evaluate the performance as a whole versus this keyword is a winner and this keyword is a looser.

(IMO) The generic keywords can help drive the traffic and the transactional and brand keywords close the sale. A person could need to click on all three groups on the way to the purchase and as long as the entire portfolio drives a good return on advertising spend (ROAS) you’re good.

This was a great statistic:
“For searchers who begin their search process on a non-brand term and then switch to a brand term, conversion rates are seven times higher than when there are only non-brand terms.”

If you have your campaign set up so the last set cookie wins credit for the conversion, the generic keywords will appear to be under performing. (But) If you have it set so the first set cookie wins, the transactional keywords will appear to be underperforming.

This was another great statistic:
“larger marketers received nearly 69% of the single click users from their more established brand terms.”

This brings me to my main question about what % of this 69% would have found the “Larger Marketers” website with or without the brand term ad?

Turning brand keywords off for a short period of time should at least give you an idea of how much or little influence your generic keywords are having on the sale.

If 69% of your PPC revenue is attributed to brand terms you could shut them off in some situations and see if the revenue drops by 69%.

That kind of test would work better if you had very few distribution channels.

In my case we experienced a 0% drop in sales as detailed in post # 8 of this thread so the keywords were seen as worthless. Ironically… Even though that was a huge % of our overall spending, our overall spending stayed about the same over time. The brand keywords stopped taking credit for the sales and we were able to manage the generic keyword performance much better.

I believe it was Mark Twain who said:
"There are three kinds of lies: lies, damned lies and statistics."

Web analytics packages feed advertisers statistics that may be “Technically” correct but may or may not be an accurate representation of “Reality”.

It’s the responsibility of the advertiser to determine if the statistics represent reality or not.

[Reality Check]

Thanks Projectphp,

Please correct me if I’m wrong or if I am interpreting you incorrectly.

It seems that you agree. If the search engines were to enable advertisers to pay a fee to place a tracking pixel in the SERPS, it would be an extremely useful and popular feature.

I didn’t say it would be cheap, just useful. I’m sure they would restrict this and make it an expensive proposition. Even if it were expensive, it might be worth paying for a tracking pixel in the SERPS for a group of questionable keywords for about two weeks or so, especially if you were spending millions of dollars per year on PPC.

It is a huge ask, but I’ll bet the first search engine who enables this feature will see their ad revenues go way up. I don’t believe it’s technically impossible. It would certainly add a new level of transparency that would legitimize the entire medium.

I’m with you on the guestimate.

Here is my basic elevator pitch whenever I’m asked about the performance of our SEM campaigns.

Relevant ads are there when people are specifically searching for them.
Ad Clicks = Visits.
An economical % of people who visit convert.
Risk of not participating in PPC is much harder to measure than the reward of participation.

Without the PPC ads, consumers will have no other navigational alternative than to visit the competition. In this situation, the best case scenario is that our brand is strong enough to dislodge enough consumers from the competitions websites to compensate for the loss of revenue generated by PPC. I believe this is a long shot.

So that’s the pitch… As you can tell… It’s not working for me as well as it used to. :)

I hit all the goals but search is still seen as a questionable medium and the numbers might not be accurately representing reality. It is a large expense. Large expenses can get a disproportionate amount of attention.

Playing politics may be the way to go. I don’t know. I’m not a very good politician.

[Chris Boggs]

Wow great thread everyone!

A couple thoughts I would like to share:

It is hard to determine how PPC conversions are directly affected when driving traffic to the same pages via multiple paths. As Ian suggested, the use of a separate domain to lead all PPC traffic to - completely blocked from spiders so as to avoid duplicate content - may help to get a better handle on ROI. This is important because if you host the page on the main domain, even without the ability to navigate from those pages directly to the rest of the site, there is always the chance that some people will cut off part of the URL to reach the rest of the website, or simply remember the root URL and come back, and then still end up converting. I kind of doubt that would happen often, though, especially in the case of keyword searches historically attributed to being later in the buying cycle and thus higher-converting.

The idea of shutting all PPC off completely for a traditionally high sales time period can be costly, but it may prove the idea that PPC is much more important than your naysayers feel. When working with smaller websites, this always works. But you have the problem of competing against other media and organic rankings, which may possibly peak and convert at a higher rate, and actually work against you.

There is software available that can track and sort practically everything, and since you are conducting a large amount of business online, your main goal should be to answer the skepticism by saying "yes it can be tracked, but it will cost $XXX." I am curious as to why you cannot compare the two, and I think it must be the "tree falling in the forest" aspect to this whole argument that Frank (Aussie) mentioned that is what is most perplexing. I and many others that have commented that we simply "know that PPC works," and that the ROI is great in most cases. Unfortunately that is not enough for you to argue without separating the apples from the other fruit. So again I have to say that driving the PPC traffic to a standalone site, especially focusing on late-cycle keyword phrases, will be the best way to determine ROI. Of course the cost of developing that site now has to be mixed into the calculation, but in my opinion judging from your stated spend level, the ROI will still be there.



OK I have rambled enough. I do also want to point you to some good tracking research available at Atlas Institute (http://www.atlassolutions.com/institute/insights.aspx)

[projectphp]

It seems that you agree. If the search engines were to enable advertisers to pay a fee to place a tracking pixel in the SERPS, it would be an extremely useful and popular feature.
No, it would be PR (public relations) suicide :)

No SE that has an ounce of sanity, or significant market share, would EVER do it. Of course it would be useful, but so would putting a toolbar on every computer. Can you really see that being an effective sell to Firefox and IE's developers?

And besides, stastics is a field that is used to creating meaning form incomplete data, so there are plenty of ways to justify numbers without absolute numbers.

I hit all the goals but search is still seen as a questionable medium and the numbers might not be accurately representing reality
All life is a guess :) No one really thinks that the many warnings we read are true or accurate, and most times we are forced to make the best guess we can.

The problem with trying to find "reality" is that you don't need it. A broken watch (the analogue kind) is right far more often than virtually any other watch in existence. Twice a day versus pretty much never. But would you rather a watch that was never right but was "about" three minutes slow, or a broken watch? The latter is, as I said, right far more often :)

If the numbers don't represent absolute reality, create an acceptable guess that uses figures that no one can argue with. If someone thinks PPC's ROI, with this "would have bought anyway" percentage is 50% less, that is you need to make twice as many sales for a given spend, then reduce the targets by 50%. Then, over time, lobby for an increase using whatever grounds you think will be deemed acceptable.

IMHO, taking the angle that SEM can't justify itself absolutely and to seven decimal places therefore it shouldn't be used is creating a criteria for "should be used" that is impossible to meet. Certainty is never something we are privy too in any marketing spend, so find a level everyone will accept, or call the bluff of those that are questioning it ("Are you seriously saying PPC results in ZERO extra sales? You are joking, right? What is your real agenda here? Do you just want to score points or something?" style).

[Reality Check]

Thanks again project PHP.

I don’t understand how it would be PR suicide to provide more transparency to advertisers. The banner ad networks do this with no PR backlash. I love it.

(IMO) It would only be PR suicide if in the unlikely event, the feature revealed that most people who thought they had good ROI actually did not and prices per click dropped like a rock to better reflect their actual worth.

I understand that people are complex and somewhat unpredictable. No test dealing with people or their behavior is going to be perfect. There will always be a margin of error.

The problem is there is no good test so far.

Even in your analogy:
If you only had one watch… How would you know if it was fast or slow if there was nothing to compare it to? It’s probably OK of it’s off by a few minutes or so but how would you know?

This is the root of the problem. With no control group, how can you perform a meaningful ROI test? How would I know what % visit the site anyway?

In post #12 I gave the following sample demonstration of how an advertiser would determine ROI. I’ll summarize it here for convince.

Let’s say our product sells on our website for $100.00
Let’s say that this price would give us a 20% profit margin or $20.00 per unit.

The average cost per click is $1.00
Our conversion ratio is 10%
Or $10.00 PPC Cost & $10.00 Profit per order.

So in this situation…
1 Million dollars in PPC sold 100,000 units and we make 1 Million in ROI.

The problem is…
If 25% of the people would make the purchase anyway, than this would drops the ROI per unit down to $5.00

If 50% of our PPC traffic would have made the purchase with or without PPC then that leaves us with Zero ROI. Any more than 50% and we are loosing money on every order.

I’ve used large round numbers to help keep things simple but the truth is we don’t operate on that large of a profit margin. Like most websites, we don’t have a 10% conversion rate so the % of people who would have visited our website anyway would negatively affect the ROI at smaller percentages than in this demonstration.

[projectphp]

"Every user tracked! - Google sells out to advertisers"

How many (milli)seconds before that hedline hits the "streets"? I reckon within half an hour there is one on MSN :)

Besides which, why is it Google's job? Do TV stations provide help measuring the effectiveness of TV ads? Is there a ratings box in everyhome? Of course not, precisely because the statistics are accepted as is. There is no incentive to do more, and enough advertisers realise the benefit of TV without needing the absolute reality.

With no control group, how can you perform a meaningful ROI test? How would I know what % visit the site anyway?
"Anyway" how? How would they get there "anyway"? That is such a loosely defined test, how can you test for it?

And even if you could, why would you need to?

I don't "need" to know which telegraph pole resulted in people coming to a garage sale, and even if I did, it wouldn't mean that is teh only pole worth sticking an ad up on. Ditto which coke ad made people want to drink it over rival Mecca cola. Advertising is, by nature, an inexact science. What is the old quote, "I know 50% of my advertising doesn't work, I just don;t know which 50%".

The goal isn't to be accurate to 10 decimal places, but rather to create a system that people accept, and that helps make decisions. That should be the focus, IMHO, not trying to do the impossible, or virtually so.

So, back to the problem at hand, reverse the criteria. Don;t ask for accuracy before testing. No one, I am sure, would ever claim that SEM results in no extra sales, they just are quibbling over how much of the margin of how many sales can be spent on SEM. They want, in essence, an ROI equation that isn;t just:
(revenue * margin) /Adspend.

So give them one :)

(revenue * margin * {INSERT A PERCENTAGE EVERYONE FEELS IS ACCEPTABLE GIVEN THIS "ANYWAY" FACTOR}) / Adspend

That number can be either conservative (1%) or aggressive (99%), or anywhere inbetween. Whatever it is, you can negotiate from there in teh future.

[Reality Check]

Thanks again,

Well… Ok… maybe Google wouldn’t be the first one to give it a try. Ask seems like they have nothing to loose. LOL.

I completely agree with you on the visit anyway factor. It seems like a long shot to me but others would disagree.

The old saying about not knowing which half of your advertising isn’t working is the problem. If 50% of our advertising isn’t working, we are certainly loosing a boat load of cash.

There is no “Acceptable %” that everyone is comfortable with. This is the entire disagreement.

I believe it’s a low number and within an acceptable range to still turn a profit but others think this % is so high that we are likely loosing money. I am outnumbered. (No pun intended)

So the debate turns to…
“Ok then… Prove it. Show us the facts behind your claims.”

[projectphp]

There is no “Acceptable %” that everyone is comfortable with. This is the entire disagreement.... So the debate turns to…
“Ok then… Prove it. Show us the facts behind your claims.”
You are looking at this the wrong way round.

Where exactly is the disagreement? 2% vs 98%? Less? more? Establish what everyone's figure is first, setup an initial "guesstimate", and then test it.

Some ideas on how to choose a number:
- If there are five peopple, ask everyone to write down teh % they believe, discount the top and botom score, average the rest, and there you go, your number :)
- if there are less people, you abstain, and simply take the average of what is left.
- take the lowest number.

No matter what you choose you need an initial %, because once you have a starting number, you can start to argue against it with rel data. I mean, if you drop the AdSpend from SEM, and revenue decreases, you have a stronger case for saying "This "anyway" buying percentage is way off. Look how revenues last month went from $X to Y, a Z% drop. I propose we change the figure to e%".

That way, you are moving in a direction, not stuck going around in circles or dreaming about the impossible (an invisble pixel on every SERP).

In my opinion, trying to prove a point that can't be proven is the hard road. Instead of going straight ahead, and hitting a brickwall, go around it :)

[Reality Check]

Hi Chris!

Thanks for joining the conversation!

I was thinking about diverting the traffic to a completely different, pre-existing website or creating a new one that would satisfy the editorial guidelines.

You and Aussiewebmaster are right about the old tree falling in the forest paradox.

The SEM Two-Envelope Paradox:

Google places two envelopes in front of you and tells you that both envelopes contain money, and that one contains twice as much as the other, but does not tell you which is which. You are allowed to choose one envelope, and to keep all the money you find inside.

You pick envelope A, open it and see that it contains $100. Now we have a choice to make. We can keep it or trade it for envelope B.

Then the finance department says, since one envelope has twice what the other one has, envelope B either has 200 dollars or 50 dollars, with equal probability. If you switch, then, you stand to either win $100 or to lose $50. Since you stand to win more than you stand to lose, you should switch.

But, another thought might occur to you. If you had picked envelope B, you would have come to exactly the same conclusion. So if the above argument is valid, you should switch no matter which envelope you choose. But that can't be right.

SEM is profitable but on the other hand… No SEM might be even more profitable because we would have gotten most of the visits anyway. But… If we weren’t doing any SEM… They would probably tell me to start a campaign. (Either way I’m screwed)

Just testing it is a tricky situation. It’s all in the idea phase but I believe that you are correct that it would be expensive to conduct a test. It would cost us to divert the traffic as well as cost us in lost sales & revenue. They may not go for it if it’s too expensive. I’ll be happy either way as long as I can return to happily going about my work.

[mcanerin]

But, another thought might occur to you. If you had picked envelope B, you would have come to exactly the same conclusion. So if the above argument is valid, you should switch no matter which envelope you choose. But that can't be right.

Under some circumstances, it can be right, ala the famous Monty Hall problem:
http://en.wikipedia.org/wiki/Monty_Hall_problem

I have no idea how this would apply to the current problem, but it's a good object lesson in logic, and how sometimes the "obvious" answer (and the logical one) can be the wrong one.

Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

Most people say that since there are 2 doors remaining, it doesn't matter if you switch or not, since the chance of being right is now 50/50. But in reality, it's 1/3 if you stay, and 2/3 if you switch. So you should always switch.

I'm tired, so like I said, I'm not certain if this can be applied here. But knowing that sometimes the answer is to switch can be useful - being able to look at a problem in a different light can also be helpful.

At the very least, it's an interesting diversion.... ;)

Ian

[Reality Check]

Thanks Projectphp.

I can’t test it simply by shutting off the campaign. The revenue from our SEM efforts is big but its fairly insignificant compared the overall sales from the website. Seasonality and competitive conditions routinely create sales spikes and dips larger than the total contribution from SEM. If I shut off all the keyword buys, the overall revenue could actually go up depending on situations beyond our control. It wouldn’t be a statistically significant test.

Trying to accurately measure the ROI from SEM can be extremely difficult unless you have a controlled environment. If our only source of revenue and traffic was SEM, no one would bother me about it.

[Reality Check]

Thanks Ian,

It’s true. Everyone wants to know how much ROI is behind door #2. :)

[mcanerin]

This is an interesting problem, so I decided to go looking outside the box. There are lots of things that are difficult to measure ROI on, so someone must have come up with some ideas, if only for a completely different industry.

That brought me to this article:

http://www.csoonline.com/read/120902/calculate.html

The fellow in this article is trying to justify the ROI for some security patch management software. His problem is that if no one tries to hack into the system then the security software has an ROI of 0, but if someone does and it's stopped, then it's much higher. Basically, he's at the mercy of the "what happens if it's not in place" scenario.

The article points out that this is a binary approach, and the wrong one for this area. they suggest that you look at it from a statistical approach, like an insurance company would, deciding on the ROI for wearing a seat belt or you dying, etc. These are not the type of things you can use a binary ROI on.

In this case, they suggest taking the approach of what would be the cost of a complete failure and divide it by the percentage change of that happening.

So, in your case, you have 2 options:

1) The person visits based on the PPC ad. Remember that Google states that if a visitor sees an organic listing AND a PPC ad on the same page, they are 3 times more likely to click - on the organic link. This should be acknowledged as part of your argument, IMO.

2) The person would have visited without the PPC anyway. For these people, the ad is wasted. This group is almost certainly smaller than the first group.

I'm not sure which formula to use at this point, but it would probably be something like 15% of the PPC conversion rate + 1/3 of the organic traffic conversion rate for terms where the PPC appears on the same page as organic listings.

I'm mathematically challenged, so I'm sure the above can be improved on. Probably dramatically.

You know what I do? I just point out that the standard spend for modern businesses with multiple advertising streams (radio, TV, print, Internet) is 15% on PPC, and that aggregate data is based on a lot more users in less volatile situations, but still the same human beings making decisions. Therefore the PPC budget should be about 15% of advertising spend for a traditional business, higher if it's more of on online business.

I know that's simplistic, but it's SO much easier, and it's based on real data, from a lot of different sources, using different analysis methods, which is enough to make most of my clients happy. The actual range I have information for is 10-25%, with the mode at 15%. In this case, the industry was the traditional travel industry (not resellers, but the actual destinations and carriers).

Does that help?

Ian

[Reality Check]

Wow! Thanks Ian!

This is great! I need to read your post and the article a few more times so I can better understand the possible implications and opportunities before I can respond in a meaningful way. I do believe you are on to something.

I completely agree about calculating ROI based on calculated risks. You are correct. Other non-marketing industries do have the same types of problems. How do they solve their problems and what can the SEM industry learn from their experiences and solutions? I just need to think about how it applies to my situation. I think it might fit.

(IMO) The air travel and destination marketing websites are excellent examples of large industries who are increasingly dependent on SEM but suffer from large spikes and dips in seasonality and competitive conditions which can make ROI calculations more difficult.

Your ideas are very thought provoking. I’ll need a day or so just to think about it.

:) Great post! Thanks again.

[AussieWebmaster]

I think we need to take a step back. PPC can be measured independently of all other factors. I do it all the time.

Launch a new website, create landing pages etc - before you start throwing banner traffic at it, before the site has any branding, begin a PPC campaign.

The only traffic going to your site will be PPC originated traffic. Measure conversion etc. and off you go.

[Reality Check]

Thanks Aussiewebmaster,

I’m not sure I follow you.

That would work great for a new website, but how would that apply to my situation? Or any preexisting, enterprise level website that does massive amounts of daily transactions?