Kyle Rush, Head of Optimization at Optimizely

When I’m not running experiments on Optimizely’s conversion funnels, I love to interact with the optimization community. GrowthHackers has one of the best communities out there and last week I hosted an Ask Me Anything (AMA). The questions were very high quality and covered topics like running multiple tests at the same time, how to overcome technical hurdles, how multi-armed bandits can be helpful, what to do with inconclusive tests, and more.

I’ve provided a summary of the topics and one answer, but if this piques your interest please go the GH thread and contribute to the conversation.

Question:

How should companies think about prioritizing their tests, and then making tests an ongoing part of their strategy?

Answer:

In terms of prioritization, it’s important to be very analytical. At Optimizely, it usually takes us about two weeks to before we make a decision on experiment results. Compared to the Obama campaign (which had more traffic) we have a pretty high opportunity cost of running an experiment so do our best to make sure it’s a good experiment. One of the best methods for determining the potential ROI of experiment ideas is to look at past experiments. If you got an increase the first time you changes your headline, but the next three attempts failed, move on from headline tests for a bit. I have found it a lot harder to find the magical small change that produces a big effect. So I would say test your big ideas first if possible. Obviously there is no statistical truth to this, but in my experience, bigger changes have a higher likelihood of reaching significance.

There are a lot of solutions to making tests an ongoing part of a strategy. I think the single best thing to do is to allocate headcount towards it. If you can swing it, hire a dedicated optimization person. If you can’t, get everyone on the team (designers, engineers, product managers) evangelized. Let everyone run an experiment and then hopefully everyone will see how mission critical it is. Lastly, when you have a winner, spend a lot of time analyzing what the effect was on the bottom line of the company. If it’s a beneficial effect, trumpet the success to the entire company. Bring it up in meetings, etc. Chances are nobody will be able to point to another change that can easily be tied, like a winning experiment, to the bottom line of the company.

Other questions:

• I talk to so many people who test a couple of buttons or headlines and then don’t know what to do/test next. What’s your advice for them? @morgan

• What is one of the most successful tests that you’ve run and what were the inputs that led you to want to run that test (or was it just a lucky guess)? @sean

• In your experience, have there been times when irrespective of the result of the test you’ve implemented something else? If yes, is there any pattern you’ve noticed to when you’ve found yourself doing this? @anujadhiya

• What do you do with inconclusive tests? Keep the new variant? Keep the old variant? Come up with another variant? How important are A/A tests in your opinion? Should they be run regularly? @morgan

• What were the key CRO lessons learned while working for Obama for America? @everette

• What’s your take on running multiple simultaneous tests on a single page? Multiple tests on the same site, but with the same goals, and overlapping audiences (e.g. test on a home page, product, category, cart, checkout all at the same time)? @peeplaja

• More of a personal question, how did you get started in optimization and what keeps you in the business? @angelolireezy

• What would you recommend to someone just starting in the world of optimization? What should they learn, read or do ASAP? @ross

• What are the gotchas with Optimizely that you feel everyone should know about? I love the software, but sometimes Optimizely feels like Optimistic-ly. @dem_z

• What are your favorite [re]sources to get CRO ideas? (And thanks for doing this AMA, it’s truly awesome to have guys like you here) @dem_z

• I do CRO for e-commerce and want to roll up many product pages into one test. We tried to make URL targeting work to show the variation for multiple product pages. Everything we tried only loads one product page on all the pages though. I tried support but maybe you can point in the right direction to understand more about my scenario. @alexdesigns

• I’m a completely non-technical growth hacker. Sometimes it gets in the way for growth hacking. How necessary do you think it should be for people like us to be skilled technically? How do you recommend starting? Any recommendations on tools to use for non-technical people? @samuellee

• Does Optimizely have any plans to add bandit testing? Are there any circumstances where you’d prefer to run a bandit test? @peeplaja

• My question is: In your opinion, what is the most important factor that leads to success in an optimization process? @dylan

Keep the AMA going. Please feel free to ask me anything.