KLM scales its test-and-learn culture with Optimizely

The best way of knowing is doing. Although KLM Royal Dutch Airlines had always been pioneering the forefront of digital developments and has always had a culture of engaging customers in developing new products and services, KLM’s web teams reached the limits of their experimentation capabilities in 2017. As the need for experimentation by internal product teams and data-driven decision making increased, KLM was looking for a reliable data-driven test partner … and found one in Optimizely.

Building minimal lovable products

Digital product teams at KLM are continuously working on improvements to the user experience. In an industry that is renown for its competitiveness - many different parties are trying to seduce the travel savvy customer - small tweaks to the user experience can make a huge difference. With over 30 million people booking KLM tickets on a yearly basis, building a product that ‘just works’ (a Minimal Viable Product) is not enough, it is about building digital products that are super easy to use and intuitive - Minimal Lovable Product.

This is no easy feat and requires a lot of user engagement, feedback and testing to implement. KLM therefore implemented Optimizely to scale the number and impact of experiments done by the different product teams. Now, a year-and-a-half after implementation, 6 product teams use Optimizely. Overall, KLM doubled the number of tests executed and on average the time to set up a test only takes half the time it did before Optimizely’s implementation.

We are catching up with Grazia Aroboleo and Joost Olieroock within KLM’s Customer Insights & Analytics team to talk about the implementation.

Optimizing the customer experience

The introduction of Optimizely within KLM just coincided with a KLM-wide program to roll out flight packages (incorporating baggage and ticket flexibility) in the booking flow. When Grazia Arboleo, Optimization Specialist within the Insights & Analytics team at KLM, was asked to AB-test a new design for the flight selection step on the KLM website with these new packages, she did not know that this would be an important turning point in the way design changes would be rolled out.

The team focused on building an Minimal Lovable Product based on flexible components. This would allow fast optimization and was the requirement for an agile rollout. Grazia helped the team to track the impact of the new design through experimentation. Despite high expectations, the experiment data showed clearly that the new design needed further optimization.

Grazia and the team analyzed the results and developed new hypotheses. Optimizely enabled them to run multiple experiments easily, iterate quickly and make step-by-step improvements to the design. Within a few months, the team ran more than 20 tests on this particular flow-always measuring step conversion and booking conversion rates. Some of the experiments delivered clear winners, others did not. As a result, conversion rates recovered and finally, the experience was rolled out to all visitors.

Prior to using Optimizely, KLM used to implement redesigns gradually, starting by serving the experience to a small geographic market. If the teams did not see drastic changes in their analytics or sales figures for this market, they would gradually expand the experience to other markets. This roll-out process could help to prevent drastic drops in measurements, but more subtle changes were hard to detect.

Accelerating the Experimentation Mindset

This first successful, experiment received a lot of attention within KLM and helped to drive the experimentation mindset further “It was crucial for us to be able to measure the differences between the old and the new versions of the booking flow and get reliable statistical data”, says Grazia.

The ease of setting up experiments in Optimizely has encouraged other product teams at KLM to engage with experimentation. “Prior we had third parties developing most of our experiments. Nowadays we have our own developers doing this. When new developers start using Optimizely, they are often amazed about the possibilities. It is also great that experimentation makes the impact of their daily work much clearer. The analysts from Grazia’s team support the product teams throughout the complete experimentation process (from plan to analysis). “The team I support is now submitting more test requests than I can analyze”, Grazia laughs and is glad that Optimizely’s Stats Engine, the statistical backbone of the platform, makes her life as an analyst a lot easier. She is proud that her product team adopted an experimentation mindset and is testing self-sufficiently as part of their agile way of working.

Data draws a detailed picture of the customer

Besides experimentation, the teams are also gaining insights from other data sources (e.g. analytics, surveys, heatmaps). Optimizely's integration with some of their tools, helps them to get a better understanding of customer needs and see if the variations in their experiments better fulfill these needs.

The integration with the Analytics tool allows KLM to automatically import experiment data for further analysis within a wider business context. Also heatmaps can be automatically tagged with the information about the A/B test variation that a particular user has seen. This way the analysts can differentiate between experiences during their analysis.

Joost brings up the project for KLM’s Corporate program to optimize the experience of business travellers by providing them with a special booking flow for corporate customers. “This a perfect example where we used insights from user feedback, analytics and experimentation in conjunction.”

Before developing the experience, the team wanted to draw a picture of this particular segment and understand how they like to engage with the site. To decide if and how to build the experience, the team ran an experiment with Optimizely to find out, if users were actually willing to disclose whether they were travelling for leisure or business by adding an option to specify the reason for travel. They also conducted a survey asking users to fill in a questionnaire about their booking preferences. The results were complemented with data from their analytics tool.

While the project has not yet been completed, the research has lead to the hypothesis that users might be more willing to reveal their reason for travel if they knew what the information would be used for and how it would benefit them.

This example shows how data from various sources of research helps to gain insights on users’ needs and preferences and how experiments help optimizing the experience of visitors.

A new way of learning

The scale and speed of experimentation with Optimizely has changed the way digital product teams at KLM are working. “Optimizely is really helping our teams to test and learn faster, making it easier to adapt to our customer needs”, says Grazia. “Testing is very important to measure the impact of changes towards our goal”, she adds.

Joost predicts that experimentation will soon be the gold standard across all teams at KLM, as it allows KLM to catch potential pitfalls before they are rolled out to all users. “Product owners now ask more and more to test every change to prove that it does what it is supposed to do - and if not, we can optimize. These learnings are crucial for delivering a convincing customer experience”.