The Myths of Business Experimentation

Recently, Optimizely and a prospective customer were engaged in a series of conversations. After a set of thoughtful meetings where the two teams discussed potential use cases for our product and the technical details of working together they set up one final meeting with the VP who would sign off on the agreement. That’s when

Written by:

Steven Schuler

Recently, Optimizely and a prospective customer were engaged in a series of conversations. After a set of thoughtful meetings where the two teams discussed potential use cases for our product and the technical details of working together they set up one final meeting with the VP who would sign off on the agreement. That’s when things got…interesting. “We have a number of people here who have worked in this space for years,” the VP said “won’t more experimentation mean the end of creativity and judgement based on years of experience?”

A probing question, but not an unexpected one. This leader is not alone, many teams come to Optimizely with a set of concerns or preconceived notions about experimentation. Harvard Business School Professor Stefan Thomke calls these the myths of business experimentation and he lists a set of common myths in his new book Experimentation Works.

At Optimizely there are three myths that we hear most often, the first is this idea that experimentation is antithetical to intuition and judgement.

  1. Experimentation will end intuition and judgement In his book Professor Thomke explains that experimentation is enhanced by experience; “it’s not about intuition versus experiments; in fact, the two need each other. Intuition, customer insights and qualitative research are valuable sources for new hypotheses, which may or may not be refuted, but can often be improved through rigorous testing.” For years business research has shown that in many cases experts are no better at predicting customer behavior than novices, so why not test every creative hunch and learn what works and what doesn’t.
  2. Experimentation can’t deliver breakthrough innovation
    The second myth we often encounter is that experimentation is good for making small, incremental changes, but not breakthrough innovations. Again, Thomke’s research addresses this concern “breakthroughs in business performance aren’t always the result of one or a few big changes, however; they can also come from the continuous flow of many smaller successful changes that accumulate quickly and can operate on customers over a long period of time.” This re-frame from “small” to “continuous” is key and is best captured in the Toyota corporations philosophy of kaizen which holds that innovative ideas don’t suddenly emerge out of the blue, but instead occur to those who have a foundation of accumulated knowledge based on a history of constant improvements.
  3. My team won’t have enough ideas to run a full experimentation program
    The third myth we often encounter comes from leaders who are concerned that their team won’t generate enough ideas, experiment hypotheses, to fuel an experimentation program at scale. This makes sense, many of the companies that are hailed for their experimentation prowess, like Amazon or Microsoft, have programs that are already so big they may seem impossible to replicate. But, the truth is that every experimentation program starts small. Thomke notes that insurance giant State Farm “runs between 100 and 200 tests annually and benefits significantly from what it learns.” Experimentation is also a standard practice at most startups, small companies with high-growth aspirations, Thomke cites a 2018 study that found that 75 percent of 13,935 startups founded in 2013 used A/B testing tools.

Overall, it’s very common for experimentation organizations to start small, augment the intuition and judgement of tenured employees and drive big innovations through a continuous process of making many smaller successful changes. Yes, it takes time to scale an experimentation practice, but many important business processes take time to implement, don’t let these myths hold you back.

To learn more about experimentation best practices, and other common myths, pick up a copy of Experimentation Works or view our exclusive Thomke Talks video series with the professor.

Read next

  • 5 ways AI helps you get market research right

    Today’s market research depends on data-driven decision-making frameworks and AI for improved organizational performance. Here’s how AI can help you.

    Read post
  • The smart retailer’s guide to agile ecommerce

    Today’s retailers need an agile ecommerce approach informed by data-driven decision-making strategies. Here’s what you need to know about these frameworks.

    Read post
  • Welcome! Celebrating our newest Optimizely Most Valuable Professionals! 🎉

    The Optimizely MVPs (a group of 70+ technical community contributors) have been busy writing code packages, how-to guides, and blog posts to help technical users get the most out of Optimizely products.

    Read post