I just received an email from a small business owner who is looking for advice on what direction he should go with the online advertising campaign.
That is a very tough question because the ad campaign is small and small things tend to be hard to measure so it’s very difficult for me to give good advice in these situations.
The problem is trying to figure out what works and what doesn’t when you only have a small sample size to work with.
Large advertisers have the advantage in these situations. If two sets of ads each got 10,000 clicks and one got 200 conversions and the other got 300. The one with 300 conversions is the winner.
On the other hand… A small advertiser might only get a few clicks per day. If two sets of ads are running for a week and each got 10 clicks but one got 2 conversions and the other got 3… It makes no difference because the result is within the margin of error. You would need to run the test for several months.
The math required to do statistic is beyond me but here are some simple situations.
- Lower margin of error requires a larger sample size.
If 90 out of 100 people agree on something, you can likely believe it. The closer it gets to 50/50, the larger the sample size you will need to overcome the margin of error.
- Higher confidence level requires a larger sample size.
If 3 out of 5 people agree on something, you can’t be too sure. If 300 out of 500 people agree, you can have more confidence in the result.
Aaron Wall had a great post on the subject a while back. He ran two identical ads against each other and one performed three times better the other. (If your total sample size for a split test is only 16 clicks, anything can happen.)
Here are some tools that may help you do the math.
On the bright side… If you’re dealing with small numbers, you don’t have much to loose by trusting your instincts and going with your gut until things start to pick up.