Imagine that you’re out hiking in the mountains and you decide you want to visit the highest peak in the park. You decide to start by heading toward the peak nearest you. Following the path, you get to the top and see that another nearby peak is higher. How many times would you climb the nearest peak without direction, hoping you’d land in the right place eventually? Do you think you’d ever find what you were looking for? Or would you collapse in a pile of sweaty, panting defeat three hours before sunset?
Unfortunately, the latter is probably closer to the truth. If you’d managed to get up higher and get a better lay of the land, or even if you had a map that laid out all the options and paths to your goal, you would have been able to work toward your solution without nearly so much suffering.
Well, in ideation, all those hills surrounding the peak would represent the local maximum, whereas that distant, lofty peak would be the global maximum. When you experiment with an eye only to the local maximum, you’re working with a refinement approach. This approach can give you a better result than you had before, but the problem with that approach is that, as you refine and refine toward an ultimate hoped-for solution, it’s possible that you will miss actually arriving at the best solution. An approach that embraces exploration is often more likely to get you closer to the best solution, and the best way to get the lay of the land is to take risks and experiment with more possible variations that create significant changes.
This is one important reason why you should employ multiple possible solutions that test your hypothesis in each experiment. When you experiment on many possibilities, treat each component as its own variation. Each one is a road that could lead to the best solution, and that solution may not be what you expected going into the experiment. The organizations most successful at optimizing find that their experiments reaching statistical significance often have four or more variations.
But this doesn’t mean that you should just throw out the refinement approach, especially after you’ve done some exploration experiments and might be close to finding a solution that works, or if you’re attempting to optimize smaller components of your site or app. Honing your experimentation with refinement can also elicit useful results in the right context. But it’s important that you not be afraid to think big and explore even drastic or surprising ideas. The beauty of experimentation is that you can always go back, with solid data and confidence, if an experiment reveals that your changes weren’t useful to your visitors.
Even if you change “enough” in your experiment, it’s important to set expectations realistically, since any experiment may not give you what you expect. Eddie Hartman, Co-Founder and CPO of LegalZoom.com, explains that accepting this failure early is important. He says,
“Look, not all experiments are gonna work. If all your experiments are working, you're not being bold enough. Optimizely is what allows me and my team to be bold. To take the risks we need to take in order to achieve the success we wanna achieve.”
As you begin to imagine the changes you’ll make in experimentation, be focused, but don’t be afraid to be bold. Bolder changes are more likely to achieve statistical significance, and experimenting with a radical change may elicit some surprising results you couldn’t have anticipated simply by testing variations on a theme.
Remember that failure is a part of experimentation. With a good hypothesis, you will get closer to an understanding of why and how an experiment failed when the inevitable happens.
Once you form a hypothesis and begin your experiment, measure your macro conversions, which are your primary conversion goals (such as purchases, revenue per visitor, or leads created), but don't forget to keep track of your micro conversions, too. These are metrics related to your conversion goals that show you're headed in the right direction. You can consider using, for example, the pageviews for each page in your conversion funnel, whether a visitor watched a product video, newsletter sign-ups, or the number of clicks on your CTA. Micro conversions often precede the macro conversion, so it’s helpful to follow this information to find out if you got an uplift in attention from the experiment, even if it didn’t result in a full-fledged conversion. More attention still indicates that the experiment had an impact, so even if it doesn’t give you the significant response you wanted, following micro conversions can at least tell you if you’re on the right track--and that’s good!
Also, as you move into this phase of building your hypothesis and begin thinking about running experiments, consider which changes are going to make the biggest impact in the shortest time. Your ability to take experiment results and use them to quickly implement a meaningful change should influence where and what you start to experiment with. Those changes that are high in both potential impact and ease of integration should be highest on your priority list. We’ll talk about this in a little more detail in the Planning course, but it’s worth considering at a high level now.