A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

​This is the second post in a three-part series about the optimization program at Hotwire, an Expedia, Inc. company.

Part 1: Organized Ideation: How Hotwire Runs 120+ Experiments Per Year

Part 2: A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings

Part 3: A Mobile Website Optimization Case Study: Hotwire’s Cars Page

Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked question at Optimizely: “What should I test?” A strong prioritization process arms a growth marketer with the answer: “Here’s what we’ll test, in what order, and here’s why.”

Last week in part one of this three-part series, we met Pauline Marol, the Lead Product Manager for optimization at Hotwire. Pauline’s prioritization process (how’s that alliteration for you?) is a framework that keeps her optimization engine running smoothly with hundreds of ideas in the backlog, and helps the very best ideas surface so they can be tested quickly.

As Pauline describes it, her prioritization “removes the emotion from A/B testing.”

“I don’t want to be the one making decisions on what [experiments] I run and don’t run because I am biased,” she says.

So just how good is this process? It’s comprehensive, rigorous, logical, and too good not to share—so here it is, in full:

The prioritization system is additive. If the idea meets a requirement, it gets one point. If it doesn’t, it gets zero. The points are summed in a spreadsheet to give each idea an overall score out of 10. Ideas are ranked according to the overall score.

Related post: Steps for Building Your Own Prioritization Framework

#BBD0E0 »