A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings
Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently-asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.” Last week, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog and helps the very best ideas surface so they can be tested quickly. As Pauline describes it, her prioritization “removes the emotion from A/B testing.”
This is the second post in a three-part series about the optimization program at Hotwire, an Expedia, Inc. company.
Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.”
Last week in part one of this three-part series, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process (how’s that alliteration for you?) is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog, and helps the very best ideas surface so they can be tested quickly.
As Pauline describes it, her prioritization “removes the emotion from A/B testing.”
“I don’t want to be the one making decisions on what [experiments] I run and don’t run because I am biased,” she says.
So just how good is this process? It’s comprehensive, rigorous, logical, and too good not to share—so here it is, in full:
The prioritization system is additive. If the idea meets a requirement, it gets one point. If it doesn’t, it gets zero. The points are summed in a spreadsheet to give each idea an overall score out of 10. Ideas are ranked according to the overall score.
Related post: Steps for Building Your Own Prioritization Framework
|Rule||1 Point||0 Points|
|Main Metric||Supports the company’s main metric – new bookings||Supports a secondary metric, like CTR or Net Promoter Score|
|Location||Tests a change to the results or billing pages||Tests a change located on any other pages|
|Fold||Makes a change above the fold||Makes a change below the fold|
|Targeting||Targets 100% of customers||Targets a subset of customers (repeat only, new only, top 50 markets, etc.)|
|New Information||Adds new information or a new element or removes an element from the page||Makes a change to the existing elements (copy, color, UI, etc.)|
|Benchmarking||Borrows from a success on family sites Expedia.com, Hotels.com, or CarRentals.com||No benchmarking best practice|
|Conversion Veins||Applies to two or more conversion vein themes||Applies to one or fewer conversion vein themes|
|Strategic Topic||Supports a strategic company goal||Doesn’t map to a company-level goal|
|Mobile||Would change an element of the mobile web experience, or encourage an app install||No mobile component|
|Opaque||Potential to increase of opaque share for a line of business (travel industry-specific value metric)||No influence on opaque share of business|
Thanks to this process, prioritizing Hotwire’s backlog of experiment ideas becomes a joy instead of a chore for Pauline. And remember, she has a queue of more than 400 experiment ideas as of this moment—a number that will continue to grow.
Pauline and her team have done an excellent job of increasing the visibility of their testing program, a challenge that plagues many optimization programs. But visibility can be a double-edged sword: high visibility optimization means that many people at Hotwire will come to Pauline with test ideas. Having a completely transparent prioritization system helps Pauline take herself out of the process and properly set expectations around what makes a good or bad experiment idea and when those ideas will get tested.
“If you come to me with an idea and it’s not live in two weeks, it’s not because it’s a bad idea—it’s because I have better things to test.”
—Pauline Marol, Lead Product Manager, Site Optimization, Hotwire
When everyone knows the rules and the optimization process is transparent, concerns around testing can fall away. Instead, Pauline gets to focus on being the FUN team at Hotwire—a hub for creativity, moving fast, driving business results, and being inspired by the opportunity to lead the travel industry.
Combine creative brainstorming, well-informed hypotheses, a robust prioritization process—and a dash of passion for testing and optimization—to follow Pauline’s lead and create a world-class optimization program of your own. Next week, we’ll take a deep dive into an experiment that produced the largest win for Pauline and the Hotwire team. Stay tuned for more!