A-B-Testing-5-Reasons

Conversion rate optimisation (CRO) is a huge revenue opportunity for many companies. If you have a business-critical website, the art and science of persuading website visitors to transact has significant upside.

What’s stopping CRO taking a more central role within the business? In this article I’ll discuss the #1 pitfall that prevents companies from realising real growth and maximising website revenue with CRO.

(In a hurry? Jump straight to the five reasons not to A/B test.)

Why A/B test?

First, some context – why A/B test? It’s about risk mitigation. Rather than making a permanent change on your website, you mitigate risk by A/B testing your hypothesis. The test results prove you right or wrong, and it’s quicker to test than make a permanent change on the website.

It’s the hypothesis, stupid

A/B testing lives and dies by the hypothesis. Otherwise, just what are you testing, anyway? Crucially, a hypothesis needs to be data-driven. In other words, data research is essential to inform your hypotheses. By extension, A/B testing doesn’t work without data-driven insight.

Data comes from tools and techniques such as Google Analytics, user testing, session replay videos, surveys, etc. Your data-driven hypothesis would look something like this (with a hat-tip to the excellent Craig Sullivan here):
1. Because we saw [data/feedback]

2. We expect that [change] will cause [impact]

3. We’ll measure this using [data metric]

If you can’t assign a hypothesis to a data finding then it’s probably not a hypothesis, it’s a guess.

CRO ≠ A/B testing

CRO is a multidisciplinary process that sometimes culminates in an A/B test. Put another way, A/B testing is a subset of CRO. This is a critical point because so many businesses are running regular, random A/B tests without using the data required to really grow revenue. A robust optimisation programme follows this simple workflow:

Measure > Analyse > Change*

*might be an A/B test, might not be. A/B testing is just one arrow in your CRO quiver.

For all the attention it gets, A/B testing is just the final link in the CRO chain. A sub-optimal A/B testing approach is random testing based on gut feel. An optimised, money-making A/B testing approach is based on data.

5 reasons not to A/B test

Having run A/B tests for household brands as well as smaller companies, I’ve seen first-hand how a data-driven testing framework can increase net profit overnight.

That said, the decision whether or not to A/B test should be taken on a case by case basis.

When wouldn’t you A/B test? Here are five common indicators that it’s not the right time:

  1. You don’t have much traffic to your website. As a result, the lack of sample size means an A/B test would never reach statistical confidence so you can’t trust the results.
  2. No-brainer insight. The button is broken, the font is illegible, poor formatting on the checkout on Android, 100% of survey participants didn’t understand the call to action. You catch my drift, something clearly needs fixing. Just do it.
  3. Insight vs testing. You have limited resource, and would be better off focusing on getting more user insight i.e. data. Instead of testing your own assumptions, it’s better to focus time and money on measuring users’ behavior and determining what they want and don’t want.
  4. Talent. You don’t have a team that knows how to design and code an A/B test, or they don’t have time to do it.
  5. Statistics. Concepts like non-binomial metrics make your head spin and you’re worried you’ll screw up the test design or misinterpret results. The risks of implementing a false-positive (where you think the test was good but it was bad!) are too scary to contemplate.

The Takeaway

True growth comes from understanding your audience to inform wider marketing activities and business operations. As the profile of CRO has increased in recent years, the technique of ‘A/B testing’ has disproportionately hogged the limelight. The reality is that successful CRO is grounded in data collection, interpretation and strategy that inform change on your website. A/B-testing is the current media darling but you need to empathise with your users to really achieve growth.

If you’re searching for a proven methodology to increase returns from your website, why not book a free consultation (30-minute 1-2-1 consultation about your website) to learn how it works.