3 A/B Testing Case Studies on Seeking The Global Maximum

Imagine you’re climbing a mountain: if your goal is to get to the top of the tallest mountain, and you don’t have a map of the range, it’s probably not a good idea just to start walking up the nearest slope. You’ll climb, and climb, and then ultimately reach some peak—and then what? Where do you move next if this peak doesn’t turn out to be the highest one?

In optimization, the term for the nearby, uphill peak is the local maximum, whereas the distant, largest peak is the global maximum. One of the most important traits in optimization is the willingness to think big. Being too complacent about the status quo can lead to too much fine-tuning or refining and not enough exploration. The “Refinement” path might lead you to miss out on the best solution that could have been discovered with the “Exploration” approach.

While refinement can lead you to a solution better than what you have today, we recommend exploring multiple alternatives that might not resemble the current state. Embrace the kind of humility and bravery required to say, “You know, the website we have today is far from perfect. Let’s try some dramatically new layouts, new designs, and redesigns, figure out which of those work well, and then refine from there.”

However, it’s not as simple as saying that one should always explore first and always refine second. The truth is that exploration and refinement are complementary techniques, and most effective when used in tandem. Often the process of using hypothesis testing for refinement produces key insights that can deeply inform the redesign. In other words, sometimes you need to get above the tree line to see where the bigger peak lies.

As the following case studies reveal, there are huge wins to be had from thinking big and being open to questioning the status quo. There are also important revelations lurking in smaller tests that can point the way to a major redesign. And sometimes testing is the only way to find true north amidst the chaos and confusion of major changes.

Break from the Status Quo: ABC Family

Disney ran an experiment using Optimizely on the ABC Family homepage.

Original

The page displayed a large promotion for a television show you might be interested in. After looking through their search logs, however, the Disney digital team discovered that a lot of people were searching for the exact titles of shows and specific episodes. Instead of taking the incremental approach (e.g., tweaking the promo image, or rotating the featured show), the team decided to reevaluate their entire approach. They created an alternative view, one that was less visual and more hierarchical, in which users can drill down through menus to specific shows.

Variation

Disney had defined as their quantifiable success metric the percentage of visitors who clicked on any part of the experiment page. Their goal was to lift this engagement by 10 to 20 percent. In fact, by being open to this big, fundamental change, they were able to affect an engagement increase of more than 600%.

Learn Your Way to the New Site: Chrome Industries

When I spoke with Kyle Duford, then Director of eCommerce at Chrome Industries, the cycling bag, and apparel manufacturer, the team was discussing a major site redesign. “We’re purposely using all of these tests to formulate how we approach the new website.”

The Chrome team discovered something surprising when they were AB testing the order of the three promotional content blocks on their homepage: the content they put in the center block seemed always to outperform the content they put in the left block.

Chrome Industries promo block test—middle block most clicked, regardless of content.

The team’s assumption was that because people read from left to right, they would explore in this manner. “This is gold,” Duford said. Now they know to put their most important promo block in the center, but the bigger lesson is that users seem to go straight for the central imagery, rather than scanning left to right. This is a valuable insight that may end up altering the entire new layout for the site redesign. “The look and feel will be completely different, but the ideas of the blocks of content that go into it are all being discovered through this process,” Duford says. “So while it’s important right now to understand how people shop, it’s more important because it’s going to inform our decisions going forward.”

Rethink the Business Model: Lumosity

Lumosity is a company that offers web-based games designed to improve users’ minds. Their business model is simple: users pay a monthly subscription fee to access games designed by neuro- scientists to promote cognitive function. Users derive the most benefit from training regularly, and boosting user engagement was an important goal. What wasn’t intuitive, however, was what the Lumosity development team did to increase this metric.

Lumosity’s scientists recommended that users train for 15 to 20 minutes a day, 4 to 5 times per week—not unlike physical exercise—although the site didn’t actually constrain users to a specific time limit. The data showed that people would stay logged in for many hours, but that over time, the frequency of logins declined, suggesting users were burning out.

The team hypothesized that limiting the amount of training a user could do in one day would improve engagement over time.

Giving users one training session a day and congratulating them on being done for the day might achieve their goal. Such a radical change initially made many people at the company nervous, including then Product Manager Eric Dorf, who feared that restricting the amount of a time a user could use the service they were paying for would frustrate the user base. “I felt like, if I’m paying for this as a subscription and I’m not allowed to train as much as I want, why would I pay for it?” he says. “I remember thinking, ‘Gosh, I hope we don’t piss everybody off.”

Trying out the new model as part of an A/B test mitigated that risk. The team ran an A/B test that set the original, unlimited training against the limited training variation.

Original Lumosity user experience—unlimited daily training

New Lumosity user experience—limited daily training

The results shocked Eric and his team. Users actually trained more over time in the new model. “The graph was so clear,” Eric said. “People were training more as a result of being limited.”

After making this discovery, the Lumosity team changed the way they position, build and sell their program. The message of daily training is the cornerstone of their communications to users. After this initial exploration, the team then subsequently used AB testing to refine the approach, finding the messages and marketing that best support and reinforce the idea of daily training.

Today, when a user completes a session, the message is, “You’re done. You can leave the site now,” Dorf explained. “It’s not like a lot of other gaming products that want you to spend all your time playing. The scientists are happy because more users are more engaged with training than before.”

Tips For Convincing Others to Test Big:

• Embrace the phrase, “it’s just an experiment.” The idea you’re suggesting is not permanent, it’s an experiment. Reinforcing this idea will help you convince others to try a new idea and set expectations if it fails, as experiments often do.
• Experiment on a fraction of traffic. If others are worried about rolling out the experiment, quell their fears by experimenting on a smaller subset of traffic. Just remember that the less traffic you have, the longer it takes for an experiment to be statistically significant.
• Find your champions. The experiment will likely be easier to implement if you have another voice rooting for it.

Have you run any big, fearless tests? Or had experiences convincing your team to run a fearless test? We’d love to hear about them! Please share in the comments below.

A version of this post first appeared in A/B Testing: The Most Powerful Way to Turn Clicks into Customers by Optimizely co-founders Dan Siroker and Pete Koomen.