How to Prioritize Your Test Ideas and Other Critical Questions
When I’m not running experiments on Optimizely’s conversion funnels, I love to interact with the optimization community. GrowthHackers has one of the best communities out there and last week I hosted an Ask Me Anything (AMA). The questions were very high quality and covered topics like running multiple tests at the same time, how to overcome technical hurdles, how multi-armed bandits can be helpful, what to do with inconclusive tests, and more. If this piques your interest, have a read through the questions and, of course, continue to ask me anything.
When I’m not running experiments on Optimizely’s conversion funnels, I love to interact with the optimization community. GrowthHackers has one of the best communities out there and last week I hosted an Ask Me Anything (AMA). The questions were very high quality and covered topics like running multiple tests at the same time, how to overcome technical hurdles, how multi-armed bandits can be helpful, what to do with inconclusive tests, and more.
I’ve provided a summary of the topics and one answer, but if this piques your interest please go the GH thread and contribute to the conversation.
How should companies think about prioritizing their tests, and then making tests an ongoing part of their strategy?
In terms of prioritization, it’s important to be very analytical. At Optimizely, it usually takes us about two weeks to before we make a decision on experiment results. Compared to the Obama campaign (which had more traffic) we have a pretty high opportunity cost of running an experiment so do our best to make sure it’s a good experiment. One of the best methods for determining the potential ROI of experiment ideas is to look at past experiments. If you got an increase the first time you changes your headline, but the next three attempts failed, move on from headline tests for a bit. I have found it a lot harder to find the magical small change that produces a big effect. So I would say test your big ideas first if possible. Obviously there is no statistical truth to this, but in my experience, bigger changes have a higher likelihood of reaching significance.
There are a lot of solutions to making tests an ongoing part of a strategy. I think the single best thing to do is to allocate headcount towards it. If you can swing it, hire a dedicated optimization person. If you can’t, get everyone on the team (designers, engineers, product managers) evangelized. Let everyone run an experiment and then hopefully everyone will see how mission critical it is. Lastly, when you have a winner, spend a lot of time analyzing what the effect was on the bottom line of the company. If it’s a beneficial effect, trumpet the success to the entire company. Bring it up in meetings, etc. Chances are nobody will be able to point to another change that can easily be tied, like a winning experiment, to the bottom line of the company.
• In your experience, have there been times when irrespective of the result of the test you’ve implemented something else? If yes, is there any pattern you’ve noticed to when you’ve found yourself doing this? @anujadhiya
• What’s your take on running multiple simultaneous tests on a single page? Multiple tests on the same site, but with the same goals, and overlapping audiences (e.g. test on a home page, product, category, cart, checkout all at the same time)? @peeplaja
• I do CRO for e-commerce and want to roll up many product pages into one test. We tried to make URL targeting work to show the variation for multiple product pages. Everything we tried only loads one product page on all the pages though. I tried support but maybe you can point in the right direction to understand more about my scenario. @alexdesigns
• I’m a completely non-technical growth hacker. Sometimes it gets in the way for growth hacking. How necessary do you think it should be for people like us to be skilled technically? How do you recommend starting? Any recommendations on tools to use for non-technical people? @samuellee
Keep the AMA going. Please feel free to ask me anything.