Why isn’t A/B testing done all the time by everyone? In this article I’ll discuss the A/B test promise, guilty secrets common to all tool providers, and how to get out of A/B testing purgatory.
There was a time, not so long ago, when marketers thought the internet was finally going to answer Wanamaker’s remark famous quip: “half the money I spend on advertising is wasted; the trouble is I don’t know which half.”
There is no greater exponent of marketing nirvana than A/B testing. But how easy it, really, to run a successful A/B test?
A/B testing – where are we?
Many companies with business-critical website already have an A/B test tool on their website. They tend to fall into two camps:
a) They’re spinning their wheels in A/B testing purgatory. They run random tests but they’re not seeing real uplift in terms of business growth.
b) They tried A/B testing but stopped a while ago due to resource and prioritisation
By now, you’re familiar with the A/B tool promise: “put our code on your website, run an A/B test, increase revenue!”
Is your A/B test tool is lying to you?
What A/B test tools don’t tell you
‘Back in the day’ (i.e. five years ago:)) most companies were using either Visual Web Optimizer or Optimizely. Now you’ve got Convert Experiences, SiteSpect, AB Tasty, Google Optimize and a dozen others. The industry has increasingly fragmented over the years. It’s a bull market.
All A/B test tools are powerful, easy to setup and use. I tell businesses that the choice of tool isn’t important. What matters is how you use them.
The challenge is twofold: leveraging data and understanding statistics. I discuss the former in my article The biggest barrier to growth according to marketers; this article is concerned with the latter.
All the A/B test tools I’ve come across are guilty of papering over the cracks when it comes to statistics.
The statistical problem starts with test duration. How long should the test last? Your A/B test tool doesn’t care! The reason to determine test duration is to stop you, the tester, from stopping the test when you feel like it i.e. when the results look good to you. That’s because results go up and down all the time:
Figure 1: test results fluctuate due to chance. Don’t randomly stop the test, pick a point in time when it will end.