The myths of business experimentation

 After a set of thoughtful meetings where the two teams discussed potential use cases for our product and the technical details of working together they set up one final meeting with the VP who would sign off on the agreement. That’s when things got…interesting. “We have a number of people here who have worked in this space for years,” the VP said “won’t more experimentation mean the end of creativity and judgment based on years of experience?”

A probing question, but not an unexpected one. This leader is not alone, many teams come to Optimizely with a set of concerns or preconceived notions about experimentation. Harvard Business School Professor Stefan Thomke calls these the myths of business experimentation and he lists a set of common myths in his new book Experimentation Works.

At Optimizely there are three myths that we hear most often, the first is this idea that experimentation is antithetical to intuition and judgment.

1. Experimentation will end intuition and judgment In his book Professor Thomke explains that experimentation is enhanced by experience; “it’s not about intuition versus experiments; in fact, the two need each other. Intuition, customer insights and qualitative research are valuable sources for new hypotheses, which may or may not be refuted, but can often be improved through rigorous testing.” For years business research has shown that in many cases experts are no better at predicting customer behavior than novices, so why not test every creative hunch and learn what works and what doesn’t.
2. Experimentation can’t deliver breakthrough innovation
   The second myth we often encounter is that experimentation is good for making small, incremental changes, but not breakthrough innovations. Again, Thomke’s research addresses this concern “breakthroughs in business performance aren’t always the result of one or a few big changes, however; they can also come from the continuous flow of many smaller successful changes that accumulate quickly and can operate on customers over a long period of time.” This re-frame from “small” to “continuous” is key and is best captured in the Toyota corporations philosophy of kaizen which holds that innovative ideas don’t suddenly emerge out of the blue, but instead occur to those who have a foundation of accumulated knowledge based on a history of constant improvements.
3. My team won’t have enough ideas to run a full experimentation program
   The third myth we often encounter comes from leaders who are concerned that their team won’t generate enough ideas, experiment hypotheses, to fuel an experimentation program at scale. This makes sense, many of the companies that are hailed for their experimentation prowess, like Amazon or Microsoft, have programs that are already so big they may seem impossible to replicate. But, the truth is that every experimentation program starts small. Thomke notes that insurance giant State Farm “runs between 100 and 200 tests annually and benefits significantly from what it learns.” Experimentation is also a standard practice at most startups, small companies with high-growth aspirations, Thomke cites a 2018 study that found that 75 percent of 13,935 startups founded in 2013 used A/B testing tools.

Overall, it’s very common for experimentation organizations to start small, augment the intuition and judgment of tenured employees and drive big innovations through a continuous process of making many smaller successful changes. Yes, it takes time to scale an experimentation practice, but many important business processes take time to implement, don’t let these myths hold you back.