Organized Ideation: How Hotwire Runs 120+ Experiments Per Year
Curious what optimization looks like at one of the internet’s top travel companies? In this series of three posts, we’ll dive into the optimized testing process of Pauline Marol, the Lead Product Manager for Site Optimization at Hotwire, a leading travel booking site owned by Expedia, Inc. In this first installment, we’ll explore how Pauline structures brainstorming by ‘conversion veins’. In the future, stay tuned for a deep dive into her prioritization framework for experiment ideas and a big mobile experiment win for her team.
Curious what optimization looks like at one of the internet’s top travel companies? This is the first post in a three-part series about the optimization program at Hotwire, a leading travel booking site owned by Expedia, Inc. In the series of three posts, we dive into the optimized testing process of Pauline Marol, the Lead Product Manager for Site Optimization at Hotwire.
- Part 1: Organized Ideation: How Hotwire Runs 120+ Experiments Per Year
- Part 2: A Method for Prioritizing A/B Test Ideas That Won’t Hurt Feelings
- Part 3: A Mobile Website Optimization Case Study: Hotwire’s Cars Page
After poring over hundreds of potential experiment ideas in an Excel spreadsheet, rating each one by potential impact, and sorting the backlog of submissions, Pauline is armed with a prioritized list of experiment ideas, ready to be built.
She pauses, mid-conversation, and declares: “I have the best job in the world.”
Pauline is the Lead Product Manager for Site Optimization at Hotwire. Her meticulous method of experiment prioritization is just one of many pieces of evidence of a strong culture of optimization at her company. She leads a testing program supported by a team of more than seven designers and developers. They are chartered with improving conversion rates on Hotwire.com’s desktop and mobile web properties by running over 120 experiments each year. This task has merited one of the company’s four KPIs and CEO sponsorship.
She’s an advocate for A/B testing and optimization both on and off the clock—she’s launching a “Test and Learn” program at Hotwire that’s dedicated to sharing best practices on optimization across all teams that have the ability to test. She also leads Optimizely’s San Francisco user group.
“The travel industry is extremely competitive—everyone is testing like crazy,” says Pauline. “I have dedicated time on my calendar to keep up with what’s going on in the industry so that we can continue to develop the best online experiences for Hotwire customers.”
Hotwire, owned by Expedia, Inc., provides hotel bookings, car rentals, and airfare to customers through a variety of digital experiences, including the web and native mobile apps. To improve bookings and customer retention, Pauline focuses on increasing both the quality and quantity of experiments her team is able to run.
How Hotwire Produces High-Quality Ideas to Test with “Conversion Veins”
Hotwire knows that the best optimization ideas come from their team members and customers. To support a well-rounded brainstorming process that incorporates input from many perspectives, Pauline draws on the experience of customer-facing teams, learnings from other Expedia, Inc. domains (Hotels.com and CarRentals.com), and the User Research team.
Pauline also orchestrated a company-wide call for ideas: Hotwire’s Big Test Challenge. The contest encouraged Hotwire employees submit their ideas for testing with gift cards and other incentives.
To help start the flow of creative ideas, Pauline helps her audiences classify their ideas into different ‘conversion veins’—thematic optimization areas that can affect the conversion rate and overall customer experience.
There are eight conversion veins that Pauline recommends to her audience to inspire and capture high-value experiment ideas.
8 Experiments that Stemmed from ‘Conversion Veins’
Each of the examples below shows an experiment created from brainstorming within the conversion vein. The experiment variations are all winners that are now live on Hotwire.com.
1. Social signal: Sometimes called social proof, these indicators show that a listing is popular or recommended by other travelers. These signals could include ratings, reviews, testimonials, or other iconography.
2. Confidence: Any change that can improve customer understanding, peace of mind, or assure the visitor they’re getting a competitive price. Improving confidence can happen at any stage in the booking funnel, from signup to search, the listing pages, badging and side rails, or checkout pages.
|No Confidence Module||
3. Price display: Changes to how prices of flights, hotels, or car rentals appear in search results, during checkout, or elsewhere in the customer experience. The team might experiment with how discounts are displayed or highlighting free amenities to increase perceived value.
4. Navigation: Experiments that affect how visitors search and consume information while researching and booking travel. This could include filters available to sort search results, navigation elements at different stages of the funnel, or how users proceed through their checkout process.
5. Simplicity: Any unnecessary elements that can be removed as part of the experiment can potentially improve the customer experience and improve the ultimate conversion rate.
6. Product info layout: Hotwire’s customers travel for a variety of reasons and have wide-ranging needs when they’re away from home. Is it important that a hotel has free internet, or is pet-friendly? What about parking or a fitness center? Now, is it best to showcase this information at the search results level, or on the listing page? What is the most helpful layout? All of these choices can be tested.
7. Visual imagery: There are few activities that inspire emotion quite like travel. Visual imagery is a powerful way to bring the emotion of travel into the customer experience—but does it help or hurt conversions? What’s the right balance of visuals to information and functionality for Hotwire?
8. Urgency: Travel borrows from its online purchasing cousin, e-commerce, in these types of experiments. What time-sensitive cues can encourage visitors to move from researching to purchasing? Countdown clocks, limited inventory, or a limited-time offer?
|Control||No Urgency Notification|
Each of these conversion veins has a strong basis in consumer psychology and understanding of UX/UI design principles. With these pillars of testing ideation at her disposal, Pauline is able to aggregate a wide array of experiment ideas and start the process of evaluating their potential.
Stay tuned for next week, when we’ll reveal Pauline’s prioritization process and tips for how to remain unbiased during the testing process.