A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings
Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently-asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.” Last week, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog and helps the very best ideas surface so they can be tested quickly. As Pauline describes it, her prioritization “removes the emotion from A/B testing.”
This is the second post in a three-part series about the optimization program at Hotwire, an Expedia, Inc. company.
Part 1: Organized Ideation: How Hotwire Runs 120+ Experiments Per Year
Part 2: A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings
Part 3: A Mobile Website Optimization Case Study: Hotwire’s Cars Page
Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.”
Last week in part one of this three-part series, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process (how’s that alliteration for you?) is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog, and helps the very best ideas surface so they can be tested quickly.
As Pauline describes it, her prioritization “removes the emotion from A/B testing.”
“I don’t want to be the one making decisions on what [experiments] I run and don’t run because I am biased,” she says.
So just how good is this process? It’s comprehensive, rigorous, logical, and too good not to share—so here it is, in full:
The prioritization system is additive. If the idea meets a requirement, it gets one point. If it doesn’t, it gets zero. The points are summed in a spreadsheet to give each idea an overall score out of 10. Ideas are ranked according to the overall score.
Related post: Steps for Building Your Own Prioritization Framework
#BBD0E0 »
| Rule | 1 Point | 0 Points |
| Main Metric | Supports the company’s main metric – new bookings | Supports a secondary metric, like CTR or Net Promoter Score |
| Location | Tests a change to the results or billing pages | Tests a change located on any other pages |
| Fold | Makes a change above the fold | Makes a change below the fold |
| Targeting | Targets 100% of customers | Targets a subset of customers (repeat only, new only, top 50 markets, etc.) |
| New Information |
