Case study: Losing Variations

Optimizely has worked with many clients who have been challenged to figure out how to manage a losing experiment. Sometimes, when you refuse to take a loss at face value and dig deeper, a losing experiment can result in valuable information that can give deep insights upon analysis.

One of our clients experienced this firsthand when experimenting with conversions based on lead form submissions. Their form was initially placed several clicks down from the homepage. The client hypothesized that placing the form directly on the homepage would help increase conversions by reducing the number of steps needed to get there.

But the results of the experiment were confounding. The client found a more than 20% reduction in the number of forms submitted as a result of the experiment! This was a shock for an experiment that seemed like it would be a slam dunk. What happened?

When they analyzed their results, the client found that the majority of visitors to the homepage were new visitors. As first-time visitors, these people were most likely in the research phase of their process and not ready to request a quote.

This got the client thinking: if they segmented their experiment and focused the next round on showing the submission form on the homepage to returning visitors, would their experiment be more successful? They adjusted their experiment and tried it again, focusing only on returning visitors.

This time, the experiment was a resounding success, with a 48% lift among this segment. Because the client didn’t entirely abandon a losing experiment and chose to dig deeper, they were able to recover a win from an experiment that was initially a loss. They learned something from their initial results and iterated those results into a new experiment focused on a specific segment who responded positively to the changes.