Customer Story

Optimizely x HelloFresh Case Study

Experimentation - A key ingredient for HelloFresh

HelloFresh is the leading global provider of meal kits, delivering fresh ingredients and carefully curated recipes directly to the doorstep of customers across the world. From its modest beginnings in 2011 the business has now expanded to 14 countries in the past 8 years. By the end of 2019, HelloFresh delivered close to 280 million meals to 2.97 million customers.

The organisation is dedicated to providing the best customer experience possible, constantly looking to innovate in order to provide fresh, healthy, and affordable food for families across the world. To drive this innovation, the business committed to experimentation as a core practice using Optimizely’s Web and App Experimentation platform across its product team.

With its substantial growth in customer base over a relatively short period of time, HelloFresh increased the size and depth of its offering to meet demand. In order to delight and satisfy its customers, they have introduced a wealth of new recipes, and provided a range of extra add-ons like wine and desserts, requiring different surcharge and payment options.

These new products and services were introduced to keep HelloFresh ahead of the game, but these changes came with challenges that the team had to address. For example, how these were promoted on the site, and how customers would engage with the updates. This is where Optimizely came in. Since 2017, the HelloFresh team have worked with Optimizely to make iterative changes to the user interface, which has allowed them to test feature edits on customers and react quickly to any feedback.

“Experimentation has been instrumental to our growth.”
Rémy Aldasoro
Rémy Aldasoro

Associate Product Director at Hello Fresh

Delivering solutions, fast

Working with Optimizely when making amends to their interface, HelloFresh has ensured the needs of their customers are addressed in real time. For example, the team found that a number of customers were reporting issues with accessibility on the platform after HelloFresh introduced some changes to its site. As the number of recipes offered increased, the existing user interface struggled to offer the great experience customers were used to.

Remy and the “Menu Experience” team were able to build a hypothesis on what may help fix these pain points, using qualitative, user generated, research to inform where the issues were. They then used Optimizely to test iterations to the platform, and used the real time customer feedback on these experiments to work out which ones they should develop out further. The ability to put the customer at the centre of its platform development, has meant HelloFresh is able to balance innovation and edits to its offering, with what its customers want and will respond well to.

For a fast-moving business like HelloFresh, innovating its offering and experimenting with the platform is a continual process in order to satisfy ever-evolving customer demand.

With the goal of creating a more satisfying menu experience, HelloFresh set to work designing a new, single-screen menu layout which gave its vast array of upselling options more visibility.

“Optimizely has removed the stress from our engineering team as they use feature flags to ensure any features they implement, across all domains and markets, can be instantly hidden or removed if needed. It gives the team complete control over launch timings, as well as the power to retract any changes if they don’t hit the mark with customers.”
Rémy Aldasoro
Rémy Aldasoro

Associate Product Director at Hello Fresh

Rémy Aldasoro, Associate Product Director at HelloFresh wanted to run experiments which compared the impact of the new design versus the original layout. The team used its Australian customer base as a proving ground to experiment with its new menu. By designing a fresh interface, and giving the vast array of upsell options more visibility with the single menu screen, the results after six weeks were impressively conclusive.

As Rémy explains: “With a sample size of 40,000 consumers, half of which trialled the new single menu experiment, there was an increase in surcharge uptake of 5 percent and upselling revenue by 7 percent.”

From food markets to new markets

The success of the single menu screen in Australia gave HelloFresh the freedom and confidence to begin a global rollout. Although rollouts on this scale are often full of pitfalls and traps, HelloFresh employed Optimizely’s feature flagging capabilities to help them stay ahead of any issues. This enabled developers, as well as Rémy, to quickly disable or enable changes to the application while live, to mitigate this risk. In today’s environment, expediting a hotfix to solve an issue once a release has gone like is not an issue that teams are willing to take.

Instead, to ensure the quality standards expected, and preserve the customer experience, changes are placed behind feature flags. This enables the product team at HelloFresh to keep a hand on major features, launch alignment, and communication allowing engineers to focus on the task of coding, rather than worrying about issues with the latest releases.

“An often overlooked benefit of experimentation is how well it can be used to manage risks,” Rémy explains. “Mistakes happen, that is just the nature of product development. But having the ability to make rapid changes is invaluable to keep our front end in line with customer expectations.”
Rémy Aldasoro
Rémy Aldasoro

Associate Product Director at Hello Fresh

In America, the sample size of the experiment was expanded to 35,000 users each engaging with the original and new approach to the menu. With an expanded user-base, HelloFresh saw an even more pronounced impact across a variety of metrics. This is exactly why it is important to rollout and measure different audiences in different geographical areas as it is difficult to predict the exact impact being made for audiences in different areas.

Another major reason for using different geographies is that experiments behave differently as menus and customer expectations, taste, and habits change. As experiments have repercussions on operations downstream, if customers do change their selection habits, then this impacts the forecasting and procurement decisions for the respective supply chains of different geographies. Every decision made on the digital side impacts operations backstage, which has a knock-on effect to supply chains, makes the experiment riskier.

As Rémy notes: “The experiment gave us a better understanding of how successful the new approach could be. Not only was it more simple, but the feature flagging from Optimizely allowed us to begin to roll this approach out to all of our geographies with very little risk. If anything became an issue, teams could very quickly identify them and revert the change with a toggle that can enable or disable a feature while an application or web page is live, rather than having to spend time and effort finding a fix.”

By increasing the average amount in each basket through experimentation with a new interface, HelloFresh ultimately achieved their goal of increased menu monetisation which is critical in driving the continued growth of the business.

A fresh approach to measurement

Throughout the experimentation process, measurement has been at the forefront of mind. As a company, HelloFresh has always had a strong belief that what you measure, is what you get. They have a deeply ingrained experimentation culture, which enables the team to let data guide its decision-making. Integral to this is the ability to consistently track KPIs & data, in order to move fast, spot trends early and react to them first. When working with Optimizely, HelloFresh made sure to identify the key metrics it needed to hit, which would be its measure of success.

The complexity of the different data sets that comes with being a subscription based service, means HelloFresh uses proxy metrics in order to track how different areas of the business are performing, from the supply chain through to the selection and cancellation rate of users. Using Optimizely, Remy and the team were able to apply specific proxy metrics to each experiment, to quickly identify the health of the new features and respond quickly if something was not working exactly as intended.

Increasing upsell performance was a key priority. This meant being able to measure how often customers were selecting the higher cost items or menu additions HelloFresh would present them with during the purchasing process. This then fed into the primary metric: net revenue. Being able to track this kind of insight helped to prove internally that experimentation was having an impact on the business’ bottom line. This was vital in communicating success to the wider organisation. As the results show, the ROI has been evident to all and has enabled Remy and his team to continue experimenting, in order to ensure innovation is matched with ease of use when customers interact with HelloFresh.

What’s next?

As the delivery of fresh food has skyrocketed in importance in recent months due to COVID-19 and consumers being at home more, brands are being forced to grapple with wildly shifting customer behaviour. To enhance its experimentation capabilities, HelloFresh is using Optimizely’s Full Stack alongside its own Menu Experimentation platform.

This will ensure that new experiments go beyond ‘painted door’ testing and explore key product market fit decisions. Testing will be established on the whole value chain, ranging from what customers see digitally, to what they receive at their front door. Through testing and experimentation, HelloFresh is continuing to monitor how different demographics are changing their behaviours and how they can be supported to continue to provide a delightful and satisfying eating experience.