Multivariate Tests Defined
Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.
Websites and mobile apps are made of combinations of changeable elements. A multivariate test will change multiple elements, like changing a picture and headline at the same time. Three variations of the image and two variations of the headline are combined to create six versions of the content, which are tested concurrently to find the winning variation.
The total number of variations in a multivariate test will always be:
[# of Variations on Element A] X [# of Variations on Element B] … = [Total # of Variations]
In the example below, the headline and image are tested in combination, with a baseline and one variation for each element generating four total variations for the experiment.
The process of running a multivariate test is similar to A/B split testing, but different in that A/B testing only tests one variable. In an A/B test, a minimum of one variable is tested to determine the effect of a change to one variable. In a multivariate test, multiple variables are tested together to uncover the ideal combination that is effective in improving the ultimate goal.
Benefits of Multivariate Testing
Using multivariate testing can be helpful when multiple elements on the same page can be changed in tandem to improve a single conversion goal: sign ups, clicks, form completions, or shares. If conducted properly, a multivariate test can eliminate the need to run several sequential A/B tests on the same page with the same goal. Instead, the tests are run concurrently with a greater number of variations in a shorter period of time.
Downsides of Multivariate Testing
The most difficult challenge in executing multivariate tests is the amount of visitor traffic required to reach meaningful results. Because of the fully factorial nature of these tests, the number of variations in a test can add up quickly. The result of a many-variation test is that the allocated traffic to each variation is lower. In A/B testing, traffic for an experiment is split in half, with 50 percent of traffic visiting each variation. In a multivariate test, traffic will be split into quarters, sixths, eighths, or even smaller segments, with variations receiving a much smaller portion of traffic than in a simple A/B test.
Before running a multivariate test, project the traffic sample size that you will need for each variation in order to reach a statistically significant result. If traffic to the page you would like to test is low, consider using an A/B test instead of a multivariate test.
Another challenge of multivariate testing is when one or more of the variables being tested do not have a measurable effect on the conversion goal. For instance, if variations of an image on a landing page do not affect the conversion goal, while modifications to a headline do, the test would have been more effective run as an A/B test rather than a multivariate test.
Examples of Multivariate Testing
Common examples of multivariate tests include:
- Testing text and visual elements on a webpage together
- Testing the text and color of a CTA button together
- Testing the number of form fields and CTA text together
Using multivariate testing as a method of website optimization is a powerful method of gathering visitor and user data that gives detailed insights into complex customer behavior. The data uncovered in multivariate testing removes doubt and uncertainty from website optimization. Continuously testing, implementing winning variations and building off of testing insights can lead to significant conversion gains.
The Big Book of Experimentation
Discover 35+ winning experiment ideas that are generating millions in revenue.
Experimentation: The What & How of Digital Competitiveness
Learn the benefits of experimenting at scale from this original research report from the Harvard Business Review.
Experimentation Maturity Model
This assessment is the starting point to understanding your organisation’s capabilities and will set you on the path to building a high-performing program.