Skip to Main Content

Optimization glossary

Multivariate testing

Multivariate testing defined

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

Websites and mobile apps are made of combinations of changeable elements. A multivariate test will change multiple elements, like changing a picture and headline at the same time. Three variations of the image and two variations of the headline are combined to create six versions of the content, which are tested concurrently to find the winning variation.

mvt_browser.png

The total number of variations in a multivariate test will always be:

[# of Variations on Element A] X [# of Variations on Element B] ... = [Total # of Variations]

In the example below, the headline and image are tested in combination, with a baseline and one variation for each element generating four total variations for the experiment.

mvt_table.png

The process of running a multivariate test is similar to A/B split testing, but different in that A/B testing only tests one variable. In an A/B test, a minimum of one variable is tested to determine the effect of a change to one variable. In a multivariate test, multiple variables are tested together to uncover the ideal combination that is effective in improving the ultimate goal.

Benefits of multivariate testing

Using multivariate testing can be helpful when multiple elements on the same page can be changed in tandem to improve a single conversion goal: sign ups, clicks, form completions or shares. If conducted properly, a multivariate test can eliminate the need to run several sequential A/B tests on the same page with the same goal. Instead, the tests are run concurrently with a greater number of variations in a shorter period of time.

Downsides of multivariate testing

The most difficult challenge in executing multivariate tests is the amount of visitor traffic required to reach meaningful results. Because of the fully factorial nature of these tests, the number of variations in a test can add up quickly. The result of a many-variation test is that the allocated traffic to each variation is lower. In A/B testing, traffic for an experiment is split in half, with 50 percent of traffic visiting each variation. In a multivariate test, traffic will be split into quarters, sixths, eighths, or even smaller segments, with variations receiving a much smaller portion of traffic than in a simple A/B test.

Before running a multivariate test, project the traffic sample size that you will need for each variation in order to reach a statistically significant result. If traffic to the page you would like to test is low, consider using an A/B test instead of a multivariate test.

Another challenge of multivariate testing is when one or more of the variables being tested do not have a measurable effect on the conversion goal. For instance, if variations of an image on a landing page do not affect the conversion goal, while modifications to a headline do, the test would have been more effectively run as an A/B test rather than a multivariate test.

Examples of multivariate testing

Common examples of multivariate tests include:

  • Testing text and visual elements on a webpage together
  • Testing the text and color of a CTA button together
  • Testing the number of form fields and CTA text together

Using multivariate testing as a method of website optimization is a powerful method of gathering visitor and user data that gives detailed insights into complex customer behavior. The data uncovered in multivariate testing removes doubt and uncertainty from website optimization. Continuously testing, implementing winning variations and building off of testing insights can lead to significant conversion gains.