Two rules about measuring online advertising campaigns
I just received an email from a small business owner who is looking for advice on what direction he should go with the online advertising campaign.
That is a very tough question because the ad campaign is small and small things tend to be hard to measure so it’s very difficult for me to give good advice in these situations.
The problem is trying to figure out what works and what doesn’t when you only have a small sample size to work with.
Large advertisers have the advantage in these situations. If two sets of ads each got 10,000 clicks and one got 200 conversions and the other got 300. The one with 300 conversions is the winner.
On the other hand… A small advertiser might only get a few clicks per day. If two sets of ads are running for a week and each got 10 clicks but one got 2 conversions and the other got 3… It makes no difference because the result is within the margin of error. You would need to run the test for several months.
The math required to do statistic is beyond me but here are some simple situations.
- Lower margin of error requires a larger sample size.
If 90 out of 100 people agree on something, you can likely believe it. The closer it gets to 50/50, the larger the sample size you will need to overcome the margin of error. - Higher confidence level requires a larger sample size.
If 3 out of 5 people agree on something, you can’t be too sure. If 300 out of 500 people agree, you can have more confidence in the result.
Aaron Wall had a great post on the subject a while back. He ran two identical ads against each other and one performed three times better the other. (If your total sample size for a split test is only 16 clicks, anything can happen.)
Here are some tools that may help you do the math.
http://www.splittester.com/
http://www.surveysystem.com/sscalc.htm
http://www.raosoft.com/samplesize.html
http://www.surveyguy.com/SGcalc.htm
On the bright side… If you’re dealing with small numbers, you don’t have much to loose by trusting your instincts and going with your gut until things start to pick up.
April 4th, 2007 at 7:49 am
Dave . . i want to say that you’ve got a very nice blog here and that i’m really jealous that you get to work out of Hawaii! To address your post, I have to agree that the benefits of large companies is that you get to see almost instantly what your marketing ideas do vs. a small site where they have to wait for days . . . I would suggest that the smaller companies that they take the time to invest some money to quickly find what works for their site as this is why it’s called research and development . . . in addition to ad testing, your client should also consider using web optimizer by google to make sure their landing page is converting as high as possible! Smart companies continuously tweak everything to make sure coversion rates at all levels are as high as possible since there are so many factors that can change rates - season (tax, holiday), types of traffic sources (google continuously adds new sites to their rotatios)!
April 4th, 2007 at 5:59 pm
Hey,
It is difficult for small advertisers to measure ad performance, but I agree with you there isn’t as much to lose. It seems like the margin of error is negligible because the ability to experiment and use unconventional advertising methods actually gives a smaller advertiser an upper hand, take internet advertising for example, many small companies were able to compete with million dollar ad budgets just using SEO alone.
April 17th, 2007 at 5:08 pm
Just found your blog, very insightful, I’ll have to subscribe.
Another great tool to test performace of ads and stop guess working is Google’s new Website Optimizer. I’ve just started using it and I love it! I’d like to hear what you think about it.
April 18th, 2007 at 4:42 pm
Interesting post, Dave. Really makes you think about how easily statistics can be manipulated to mean different things based on the vulnerability of others\’ perception. Numbers are quite malleable. $12.99 vs. $13.00, \’nuff said. A good argument for turning off ad optimization in Adwords when split or multivariate testing your ads, though.
April 20th, 2007 at 2:18 am
Google has a new (free) tool that may help smaller businesses to address the problem of what works on their website and what doesn’t. It’s called Web Optimizer (Beta). Might be worth a try.
Here’s the link:
http://services.google.com/websiteoptimizer/
Click on the link for the demo, which explains what it does and how it works.
(No, I don’t work for Google — just love many of their free tools!)
Aloha.
May 19th, 2007 at 12:01 am
hi, Dave, great post!
just to follow up - another good tip for people dealing with smaller samples is to be as specific as possible.
for example, if you’ve got a keyword ad on ‘broad match’, related terms will fire that ad in addition to the keywords you specified. that makes it more difficult to draw conclusions.
another way of doing this is to target ‘long tail’ terms. if someone clicks on an ad against the keywords “buy 5th grade math textbook today”, you can be surer of their motives than someone who clicks on an ad running against “math textbook”.
daniel
June 10th, 2007 at 5:19 pm
Hey Pua,
I think that’s a great idea. One strategy is to integrate a blog into a website somewhat seamlessly in an effort to use the blog as bait to get people to visit the main website. That type of strategy is much harder to do when you use something like Blogger.
About.com is a good example. They have about 500 sub-domains. The home page of each sub-domain is a WordPress blog.