TL;DR

• Optimizely’s most successful landing page A/B test was a three-word change that made our Pay Per Click landing page “symmetric” with our SEM ads.
• Visitors who saw the symmetric version of our PPC landing page converted to sales leads more frequently than users who saw the non-symmetric version, with high confidence.
• The observed improvement in conversions from visitors to sales leads was more than 39%.
• This symmetry experiment saved Optimizely hundreds of thousands of dollars and led to tens of thousands of additional sales leads.

Background

You may have heard that keeping your ad consistent with your landing page is good. But is that a myth? And if it isn’t, how can you quantify the impact of this symmetry? In early 2013, we performed an experiment on our Pay Per Click landing page aimed at answering these questions. Our goal was more to answer an interesting question than to find a conversion lift, but we were pleasantly surprised to discover that making our landing page mirror our ads led to a tremendous lift in conversions.

The experiment

We send traffic to our PPC Landing Page from lots of advertising campaigns, including search marketing campaigns. In some of our SEM ads we call ourselves an “A/B testing tool” and in others we call ourselves a “Split testing tool” and so on. These are all valid ways to describe Optimizely, but some people think about our space differently.

In the control, the headline of the page was always the same. It always said “Test it out for free,” no matter which ad the user had clicked on.

Click to view in a new window.

In the variation, the headline of the page mirrored the search ad. For example, if the search ad said “A/B testing tool” then the headline on the landing page would say “A/B testing tool,” and if the search ad said “Split testing tool” then the headline would say “Split Testing Tool.”

Click to view in a new window.

Original vs. Variation

Those are just a few examples, but conceptually this is how we split the traffic between control and variation.

Click to view in a new window.

Experiment results

We saw a 39.1% increase in conversions from visitors to leads in the variation with symmetric messaging. In the control, which had non-symmetric messaging, the conversion rate was 12.21%. But when the headline on the landing page was symmetric with the ad copy, that conversion rate jumped to 16.99%.

Click to view in a new window.

Funnel impact

We get a sales lead whenever someone submits a form on our PPC landing page, so the 39.1% increase in form submissions translated to a proportional increase in sales leads. We didn’t pay a penny more for the 39.1% increase in sales leads. In fact we paid slightly less because the experiment reduced the bounce rate on the landing page, thereby improving our Quality Score in AdWords and giving us a discount on a CPA basis.

We came into this experiment hoping to answer an interesting question and ended up getting tens of thousands of additional leads at no added cost — leads that would have cost us in the hundreds of thousands of dollars had we not run this test. (We keep our marketing spend, lead numbers, and cost per lead data  pretty close to the vest, so please forgive us for the vagueness here!)

Common Questions

Q. What exactly do you mean by symmetry?
A. There are different kinds of symmetry, but in general two things are symmetric if there is something you can do to them that leaves them unchanged (source). In this case, what makes the landing page and the ad copy symmetric is that you can replace the headline on the landing page with the headline in the ad text and they will remain the same.

Q. Did you repeat the test and notice similar results?
A. Yes, we repeated the test and noticed similar results — not only on this landing page, but in other parts of the marketing funnel as well. We also have a long-term A/A test on this page to help detect any potential anomalies in the data.

Q. Were the additional leads lower quality?
A. The truth is that we don’t know. When we ran this test in 2013 it was not possible to measure that. We can say, however, that (1) we didn’t see any evidence that this is the case, and (2) we now have an integration with Bizible that helps users measure differences in lead quality by variation.

Q. Is 39.1% the best result Optimizely has had on an experiment?
A. Not at all. We’ve had experiments that have led to statistically significant increases of more than 200%, but those experiments were for other conversion goals. This is the experiment that had the biggest impact on our sales leads.

Q. Do you have data around the impact of symmetric messaging in other parts of the marketing funnel? Or do you know only that it makes a difference when it comes to text ads and landing pages?
A. Yes, we have data that demonstrates symmetric messaging leads to statistically significant improvements in other parts of the marketing funnel. We may release that data in a future blog post.

A special thanks to Cara Harshman for encouraging me to write this post, and for her contributions to it!