Multivariate testing (MVT)

Multivariate testing uses the same core mechanism as A/B testing, but compares a higher number of variables, and reveals more information about how these variables interact with one another. Think of it as multiple A/B tests layered on top of each other.

As in an A/B test, traffic to a page is split between different versions of the design. The purpose of a multivariate test, then, is to measure the effectiveness each design combination has on the ultimate goal. Once a site has received enough traffic to run the test, the data from each variation is compared to find not only the most successful design, but also to potentially reveal which elements have the greatest positive or negative impact on a visitor's interaction.

Advantages

  • Multivariate testing is a powerful way to help you target redesign efforts to the elements of your page where they will have the most impact.
  • This is especially useful when designing landing page campaigns, for example, as the data about the impact of a certain element's design can be applied to future campaigns, even if the context of the element has changed.

Limitations

  • The single biggest limitation of multivariate testing is the amount of traffic needed to complete the test. Testing too many elements at once can quickly add up to a very large number of possible combinations.
  • Sometimes it's not worth the effort to run an MVT test when a few well-designed A/B tests will suffice.