North America Optimizer Award Winner - 2022

BEST EXPERIMENTATION PRACTICE

This award is presented to the Optimizely customer who runs stellar experimentation programs, demonstrating strong data-driven approaches to solve real customer problems and drive impactful business decisions.

SeaWorld and the digital opportunity

SeaWorld Parks & Entertainment Inc. is a leading theme park and entertainment company that connects guests of all ages to the natural world through up-close experiences with animals, educational attractions, thrilling rides, and spectacular shows. 

As one of the world’s foremost zoological organizations, SeaWorld is committed to the conservation of wildlife and natural habitats worldwide, and have rescued over 40,000 animals to date. Its portfolio of brands includes well-loved parks such as SeaWorld®, Busch Gardens®, Aquatica®, and Sesame Place®. 

As its customers have increasingly turned to digital channels to purchase tickets and add-on experiences, SeaWorld, as part of a larger digital transformation initiative, took on the challenge of modernizing their 12 branded websites, with a relentless focus on improving usability and increasing conversion. 

Identifying opportunities through a comprehensive data framework

Historical website data was sparse, so the newly organized Digital team established a baseline of tools, methods, and KPIs to guide the optimization effort: 

• Granular conversion funnel reporting to identify significant drop-off points 
• Customer feedback study to understand top detractors and promoters 
• Page-level interaction analysis and heatmaps to recognize on-page frustrations and engagement 

• Live usability studies and session replays to gather qualitative insights and identify root causes 
• Transaction data analysis to determine impactful segments to focus on 
• A/B testing to validate hypotheses independent of other business changes 

Over 100 hours of research and analysis led to several actionable insights: 

• Over 75% of site traffic was on a mobile device, and because the sites were not designed or optimized for mobile, user frustration was much higher there than on desktop. To have the greatest impact, they’d need to take a mobile-first approach.  
• User research showed that customers lacked clarity at key moments in the purchase flow. Questions like “Do younger children get in free?” and “What if I need to reschedule or cancel a planned visit?'', without an easy way to find an answer, led to uncertainty, support calls, and lost revenue. It was important to learn what customers need to know, and when, and then placing that information where it mattered most.  
• Session replays illuminated user behaviors and expectations throughout their journeys. It became clear that users wanted to click on images to get to more details, poor form design caused friction between the cart and checkout, and checkout errors became blockers due to unclear error messaging. 

Relentless pursuit for optimized experiences through rapid experimentation

SeaWorld identified Optimizely as a powerful experimentation partner due to the simplicity of using a visual editor, robust post-experiment analytics, and ability to scale support for multiple websites and platforms including native mobile apps. To aid in SeaWorld’s success, Optimizely’s Experimentation Services team leaned in to validate some of the early metrics, allowing SeaWorld to double down on a solid foundation to scale rapidly. SeaWorld also integrated Optimizely with Quantum Metric to sync real-time customer behavior with their existing testing strategy which proved critical to provide direction and context. This integration allowed the group to focus experimentation on top customer needs, improved the testing success rate, and ultimately validated the impact of optimization on the customer experience.

The SeaWorld team wanted to maximize the ROI on early experiments, so their experimentation journey started at the bottom of the funnel, where conversions occur. Risks were mitigated by testing hypotheses on their lower traffic sites, and porting validated solutions to higher traffic sites. 

Over the course of a few months, Optimizely’s high-velocity testing capabilities enabled SeaWorld to run over 50 experiments from homepage to checkout. Successful experiments informed SeaWorld about what worked best for their customers. As is often seen with experimentation, some findings were contrary to prior beliefs, reinforcing the need for frequent and consistent A/B testing. 

SeaWorld’s team was new to the company, but they brought meaningful previous e-commerce experience and relied on best practices during experimentation: 

• Every brand and park website had different customers, products, and associated behaviors. As a result, the team adapted to new findings for each website without assuming winning experiments were perfectly transferable to other sites. 
• With the goal to drive impact quickly, the team started with compound changes on a single page. If there was an additional need to understand the impact of individual changes, they would be decoupled and validated separately via multi-variant testing. 
• Because the business runs various sales/promotions at different times of the year, winning experiments were re-tested multiple times to ensure results were consistent during promotional events. 
• Optimizely data was tied to other sources of input including customer feedback, session replays, transactional product mix – this led to holistic data-driven decision making that went deeper into why some experiments worked while others did not. 
• In some cases where trade-offs were required, in-session experimentation data was supplemented with offline/lifetime value data to understand the downstream impact of a change and determine if it was net-positive for the company. 

An example of a cart optimization experiment that drove impact:

Analysis and research

• Funnel data showed cart-to-checkout progression rate was lower than benchmarks, especially on mobile devices
• Session replays and heatmaps showed users weren’t scrolling past 60% of the screen, therefore missing important components such as cross-sells. Additionally, there were several unintentional clicks: eg, on the header, on the CTA to go to checkout before the login form was filled.
• Transactional data showed only some users were purchasing products that required sign in; signed-in users converted lower vs. guest checkouts.

Hypotheses

• Separating the cart flow into two distinct steps (view cart, login), and redesigning the form will reduce cognitive overload and correct user errors
• Simplifying the order summary and swapping its placement with the list of items will allow users to manipulate cart items but also see cross-sells
• Defaulting to guest checkout will reduce the friction for majority of users
• Displaying a guest-friendly cancellation policy will increase confidence
• Increasing the size of buttons and adopting a modern design system will bring the experience closer to other websites that users are familiar with

Results (vary based on website)

• 5-10% increase in cross-sells added to cart
• 3-4% increase in visits from cart to checkout
• 1% lift in conversion

 

Before (Control)	After (Test)