A/B Testing -a primer

a weapon in the armoury of the modern marketer

“Experimentation is the least arrogant method of gaining knowledge.” — Isaac Asimov

As a modern marketer it is essential to utilize data to your advantage and user experience research is gaining more importance day by day. One of the most important methods of User Experience is A/B testing. It is essentially a tool to improve designs or codes and choose the best version of the elements which leads to maximum visitor count.

What is A/B Testing?

A/B or A/B/n testing is a technique where different versions of the same webpage are compared to determine which one performs better. The different versions are used randomly with different users across different time frame. If the testing is conducted on two versions, it is known as A/B testing. One can perform it on multiple variations and these experiments are known as A/B/n testing. It is also known by some marketers as split testing or bucket testing. Such form of splitting exposure of different versions ensures that guesswork is taken out of the website optimization and data-driven decisions are given priority in designing user experience.

Terminologies

The two major components involved in A/B testing are:

The Control: the control is the current page that will be updated

The Challenger: this is another version of the control page with variations in certain elements and would be tested against the control to determine whether the changes affect the customer engagement or conversations.

If the user engagements increase with the use of challenger, then the challenger will take over the control and it will be recommended. The control wins if the challenger is unsuccessful in generating better engagement and conversations.

Types of A/B Testing

There are multiple type of A/B testing which can be used by marketeers according to their convenience and choice of testing elements.

Split URL testing

Different versions of the same page are hosted on different URLs and the A / B testing is run on separate URLs. The traffic is split randomly to get better data and performance of both the URLs. The performance is tracked to find the winner. At NAVIO, we run multiple Split tests to find better versions of the page using the best tools in the industry and thus optimize the design of your web pages.

Multivariate testing

It can be considered to be an extension of the typical A/B testing where only one variable is tested. Here one can have multiple variations in different elements or variables. While in typical A/B testing, only a submit button could have been tested for color in multivariate testing, the button can be tested for size, shape, color and other variations, etc. It should also be noted that multivariate analysis works best only when the website has a decently high traffic.  

Multi-Page testing

A/B testing can also be done to understand the user’s interaction of the elements across the journey on multiple pages. To understand content tonalities, sales strategies, etc multi page testing is used. It is also known as funnel testing due to its usage in understanding the sales and funnel and critical points of customer journey. It is actively used in ecommerce websites.

What are major challenges in A/B Testing?

Knowledge of Elements to be used for A/B Testing

One of the prime reasons for companies reframing from using A/B testing is the basic knowledge of it. Not knowing of what to test and how to test it can lead to wastage of time and marketing budgets. Making educated call is a matter of experience and it is required to consider existing trends and look beyond conventional techniques.

Setting up correct parameters

Many a times marketers struggle with the lack of guiding principles for selecting a sample size for their tests. To understand the sample size, one needs to have a decent understanding of statistics and data analytics. Choosing the correct sample size is one of the primary starting objectives of A/B Testing. Another important parameter is to structure the correct hypothesis. Having a deep understanding of the website data and understanding leakages in customer journey helps in creating a correct hypothesis. One should not guess the hypothesis but rely on data and research.

Culture of Testing

It is important that the research is governed independently and without too much intervention from the business leaders. As this would result in biased results.  Testing should be encouraged and the data has to be in the right place.

A/B testing is more about understanding customer journey leakages and a lot of times marketers need to check various elements for better Conversion Rate Optimization (CRO). It is an iterative process and not a single-run program. While Google Analytics and other analytics tools do help in giving a lot of information, it is imperative that one needs to understanding deeper level interactions to get the best conversion rates. A/B testing helps in reducing website bounce rate, validate website changes, improve user experience, convert more of your recurring visitors. To run successful research, one should follow a systematic framework –

1.    Brainstorming

2.    Creating page variants

3.    Running the test

4.    Promoting your winner

5.    Start the new test for better optimization

A/B testing is a process that should be continuously utilized for better optimization of marketing strategy. Utilization of different tests on a case-by-case basis.

Thus to summarize A/B testing is the practice of using randomized experiments for making business decisions. It is not trying multiple strategies in an ad manner and comparing results. A/B testing blends quantitative and qualitative methods to check content quality. It incorporates your contacts to decide the message your email campaigns will convey. It provides you with a platform to experiment and understand the market trends that influence your contacts.

Best Practices

Test everything - Is your email on trend? Will it work for your contacts? Will it

convert or flop? Testing every element of your campaign will give you insights

into your audience and will let you improve your campaigns. What works for

your competitor might not work for you, so test away!

Test in advance - An A/B test should be your first step when considering

a change to your email campaign. There is no point in going too far with a new strategy until you find out whether it’s likely to improve your results.

Trust the results - If the test results are not what you expected, resist the

urge to listen to your instincts. A controlled A/B test is always more accurate

than an opinion.

Learn from results - When an A/B test gives you a clear winner, your first step

is to send the winning campaign to the remaining contacts in the list, but

don’t stop there. Now that you know what works best for your contacts, you

can apply the changes to other campaigns as well.

Keep testing - One test is not sufficient to achieve maximum results. You

need to keep experimenting to keep up with your contacts. There’s no such

thing as a perfect email campaign; every campaign you send can be better

than the previous one.