KLM scaling its test-and-learn culture with Optimizely

Building Minimal Lovable Products

Digital product teams at KLM are continuously working on improvements to the user experience. In an industry that is renown for its competitiveness - many different parties are trying to seduce the travel savvy customer - small tweaks to the user experience can make a huge difference. With over 30 million people booking KLM tickets on a yearly basis, building a product that ‘just works’ (a Minimal Viable Product) is not enough, it is about building digital products that are super easy to use and intuitive - Minimal Lovable Product.

This is no easy feat and requires a lot of user engagement, feedback and testing to implement. KLM therefore implemented Optimizely to scale the number and impact of experiments done by the different product teams. Now, a year-and-a-half after implementation, 6 product teams use Optimizely. Overall, KLM doubled the number of tests executed and on average the time to set up a test only takes half the time it did before Optimizely’s implementation.

We are catching up with Grazia Aroboleo and Joost Olieroock within KLM’s Customer Insights & Analytics team to talk about the implementation.

Accelerating the Experimentation Mindset

This first successful, experiment received a lot of attention within KLM and helped to drive the experimentation mindset further “It was crucial for us to be able to measure the differences between the old and the new versions of the booking flow and get reliable statistical data”, says Grazia.

The ease of setting up experiments in Optimizely has encouraged other product teams at KLM to engage with experimentation. “Prior we had third parties developing most of our experiments. Nowadays we have our own developers doing this. When new developers start using Optimizely, they are often amazed about the possibilities. It is also great that experimentation makes the impact of their daily work much clearer. The analysts from Grazia’s team support the product teams throughout the complete experimentation process (from plan to analysis). “The team I support is now submitting more test requests than I can analyze”, Grazia laughs and is glad that Optimizely’s Stats Engine, the statistical backbone of the platform, makes her life as an analyst a lot easier. She is proud that her product team adopted an experimentation mindset and is testing self-sufficiently as part of their agile way of working.

Data Draws a Detailed Picture of the Customer

Besides experimentation, the teams are also gaining insights from other data sources (e.g. analytics, surveys, heatmaps). Optimizely's integration with some of their tools, helps them to get a better understanding of customer needs and see if the variations in their experiments better fulfill these needs.

The integration with the Analytics tool allows KLM to automatically import experiment data for further analysis within a wider business context. Also heatmaps can be automatically tagged with the information about the A/B test variation that a particular user has seen. This way the analysts can differentiate between experiences during their analysis.

Joost brings up the project for KLM’s Corporate program to optimize the experience of business travellers by providing them with a special booking flow for corporate customers. “This a perfect example where we used insights from user feedback, analytics and experimentation in conjunction.”

Before developing the experience, the team wanted to draw a picture of this particular segment and understand how they like to engage with the site. To decide if and how to build the experience, the team ran an experiment with Optimizely to find out, if users were actually willing to disclose whether they were travelling for leisure or business by adding an option to specify the reason for travel. They also conducted a survey asking users to fill in a questionnaire about their booking preferences. The results were complemented with data from their analytics tool.

While the project has not yet been completed, the research has lead to the hypothesis that users might be more willing to reveal their reason for travel if they knew what the information would be used for and how it would benefit them.

This example shows how data from various sources of research helps to gain insights on users’ needs and preferences and how experiments help optimizing the experience of visitors.