A/B Testing: How to start running perfect experiments and make data-informed decisions

Experimentation is not just about running tests; it's about learning from those tests and driving change. By testing new ideas and approaches, organizations can learn what works and what doesn't, and make data-driven decisions to improve processes, products, and services.    

Building a culture of experimentation brings tremendous value to companies. Harvard Business School did a study where they looked at the value testing provided to startups, especially in the ecommerce industry. They found that investors were willing to invest 10 % more dollars into companies that were experimenting than those that weren't.

Similarly, Optimizely’s research found that the media companies using A/B testing reported 9 % more digital revenues. So, we know from the data that when people experiment, they make smarter decisions.

For example, let's say you make 50 product changes by the end of the year, all of which have an effect. Some might be better; some might be worse. If you experiment, you know not to implement the non-performing changes. You're able to roll those back while getting the benefits of changes that worked well.       

So, it is helpful to prioritize your tests and play the odds better. However, for folks, it can be a scary prospect because it exposes them to the things that they did wrong. This is where the role of leadership becomes critical.    

The Role of Senior Leadership   

In some organizations, people may be hesitant to experiment or share their ideas if they feel like they don't have the authority to do so. This can be especially true if senior leadership is not involved in experimentation.      

Let's break the myth first. When it comes to experimentation, the highest-paid person's opinion doesn’t always count the most. In fact, giving too much power to the senior leadership can sometimes hinder innovation and creativity, as people may be less willing to take risks or try new things if they feel like they're being micromanaged.

Try creating a culture of experimentation that encourages everyone to participate and contribute their ideas.     

So, in pursuit of the greater good for the business and increasing your confidence level, you must be very critical and honest about the things you're doing that are not having an effect. It’ll help your business become more agile and responsive to change, which is critical in today's fast-paced business environment.   

The need for well-designed experiments   

In school and college, we all saw examples of all these great physicists and classical researchers who ran these beautiful lab experiments. They made one tiny tweak, they measured that with certainty, and even after decades we all benefit from and talk about it in the classrooms.      

However, in the business world, people over-analyze the need for experimentation and testing as an idea’s value expires the moment the consumer base shifts and devices become different. If you make a super tiny tweak and it expires in a couple of years, the effect you're having is quite minimal.      

For example, someone in your team might think highly about your website’s homepage or landing page. The Call-to-Action (CTA) on the page is trying to drive people to sign up for events or to help them purchase the products, so no point in changing too many things at once.

Because if you change the text and the color and the size of the text and the placement for a test run, it’s hard to know exactly which of these levers made the change.       

However, the problem with that type of mentality is it'll work for an organization that's running tens of thousands of experiments and can measure every tweak one by one. But most companies that are running maybe 100 tests a year, need to take big, bold leaps if they want to get somewhere.

Stop holding yourself to this idea that you must be extremely scientifically rigorous, only make very tiny tweaks, and need to be very exact with every test.    

Designing a great experiment means exposing yourself to risk, making larger changes, and doing things that move the needle substantially in terms of statistical significance.  

The science of a good experiment   

Let's say your company decides to add new filters to the product pages as a new feature. An engineer goes out, builds the code to create a filter, and gets ready to implement it on the top of the page. There’s only a single version of that filter. If it fails, we don't know if visitors don't want filters or if the usability of that filter is just poor.   

Therefore, great if you want to have a filter, but have different versions of it. You can try it on the top of the page, on the left-hand side, and in other places. You can have it fixed or floating, and even change the order of the filters as well.  

The benefit of this experiment is, once you've run this test, let’s say all variants of your filter lose. Now you know conclusively filters are not necessary for your customers. It is time to focus on something else. Or if a version of filters you tried wins, you only implement that quickly. Simply running one filter without any alternatives can lead to misinterpretation of results.    

To get the most value from multivariate testing, approach it in a structured and systematic way. It involves:   

1. Defining a hypothesis   

Before you start experimenting, have a clear idea of what you're testing, your target audience, and what you hope to achieve. Define a hypothesis in your template - a statement that describes what you expect to happen as a result of your experiment.  

2. Designing the experiment   

Once you have a hypothesis, you need to design an experiment that will test it. It involves identifying the variables you'll be testing, calculating sample size, and determining how to measure the results.   

3. Running the experiment   

With the experiment designed, it's time to run it. This involves implementing the changes you're testing and collecting quantitative data.   

4. Analyzing the results   

Once the experiment is complete, it's time to analyze the A/B test results. This involves looking at the data you've collected and determining whether your testing hypothesis was supported or not.   

5. Iterating and learning   

Use what you've learned from the experiment to iterate and improve your approach. It means using the data to make informed decisions about what to do next and continuing to experiment and learn as you go.   

How to start your experimentation journey  

Data is critical for measuring the impact of your experiments and making data-driven decisions. It's important to have a clear understanding of the metrics you're using to evaluate success and to measure everything you can to get the most value from your experiments.  

Firstly, let’s see what to avoid. When most people start with experimentation, they assume it is about making a simple tweak.

For example, if we the change color from red to blue, this will psychologically trigger the number of visitors to purchase more and increase the conversion rate. And the beauty of a button color test is if it wins, you make money, and if it loses, you lost maybe 15 minutes of your time. It's very easy to run.

But to have a meaningful effect on user behavior, you need to do something very fundamental that's going to affect their experience and deliver a significant result.  

For most businesses, experimentation is often very much on the periphery of the decision-making, so it's somebody who's just there to pick the coat of paint on a car that's already been fully designed and assembled. Or it is something driven by senior leadership. VPs and C-level executives are making all the calls, and there's a team on the ground that's merely forced to act out what they're asking for, but then they have the freedom to experiment.   

Great experimentation is a marriage of all of these. A place where people have the right to make tweaks. They have the right to be involved in the design of the vehicle itself, and they are a partner to senior leaders in that decision-making process. Senior leaders come with great ideas, and they're allowed to augment them. They're not merely there to execute and measure the ideas of others.  

Follow these steps to get going: 

1. Start small  

Don't try to change everything at once. Instead, start with small experiments that can help you learn and build momentum in real time. 

2. Focus on the customer  

Experimentation functionality should be focused on delivering value to the customer. Make sure you're testing ideas that will have a real impact on their experience.

3. Measure everything  

To get enough data and value from your experiments, it's important to measure everything you can. This means tracking not just the outcomes, but also the process and the baseline metrics you're using to evaluate success. 

4. Create a culture of experimentation  

Finally, it's important to have an A/B testing tool that uplifts the state of experimentation and innovation. This means giving people the freedom to try new things, rewarding risk-taking, and celebrating successes (and failures) along the way.  

Finally...  

A testing program is a critical tool for driving digital transformation and Conversion Rate Optimization (CRO). By building a culture of experimentation, organizations can learn what works and what doesn't, and use that knowledge to drive change and deliver a top-notch user experience to their customers.

For a step-by-step guide to digital experimentation, check out the Big Book of Experimentation. It includes best practices for web pages, case studies, and practical tips for success.