Blue Apron validates products faster with experimentation

Blue Apron embraces experimentation to deliver better experiences

Blue Apron’s core product is a meal experience that customers create with the original recipes and fresh, seasonal ingredients that are included in every box. All recipes are accompanied by printed and digital content, including how‑to instructions and stories of suppliers and specialty ingredients. Blue Apron also sells wine, which can be paired with its meals, as well as a curated selection of cooking tools, utensils, and pantry items, which are tested and recommended by Blue Apron’s culinary team.

Blue Apron has developed an integrated ecosystem that employs technology and expertise across many disciplines. The company’s supply-demand coordination activities—demand planning, recipe creation, recipe merchandising, fulfillment operations, and marketing—drive its end-to-end value chain.

We can now get an early signal from the Optimizely Results Dashboard within a few days, whereas before it would have taken weeks or longer for someone on our analytics team to pull the data and understand what was going on.

John Cline

Engineering lead, Growth/Member Experience, Blue Apron

Getting started with experimentation

Since their inception, Blue Apron has been a data-driven, customer-focused organization, and because the category of online food is growing and constantly evolving, experimentation is in the company’s DNA.

In the early days, Blue Apron used Optimizely Web for testing. It was easy for non-technical users to create and launch experiments without engineering support, and the tool also worked well with their single-page application. But there were some drawbacks. Optimizely Web only worked with client side changes on the web, which limited Blue Apron’s ability to run experiments to optimize their mobile apps or back end applications.

Deciding with confidence

Just when John and team were weighing options and solutions, Optimizely released their new offering, Optimizely Full Stack. The solution targeted Blue Apron’s precise experimentation use case and provided mobile and server-side SDKs for testing within their code base.

With Optimizely Full Stack, Blue Apron gets statistically valid experimental results on their server and mobile applications which helps them make faster, more informed business decisions. With Optimizely’s real-time results, Blue Apron was able to avoid launching a redesigned reactivation flow, which would have resulted in a significant revenue loss.

Blue Apron can now quickly and easily build and set up new tests that were simply not possible before.

Reminding new users of upcoming deliveries: A hypothesis

Because of Blue Apron’s unique sourcing and fulfillment model, customers must decide whether or not to receive an upcoming order shortly after their previous box arrives. While this system works well for experienced customers, new users often require more hand-holding. They noticed that new customers were more likely to miss their order management cutoff point in their second week. Blue Apron hypothesized that customers that received these “surprise boxes” had poorer retention dynamics, and consequently poorer lifetime values than those who avoided surprise boxes through proper subscription management.

They tested reminding new users that their cutoff was approaching, and encouraging them to manage their upcoming deliveries to see if it would decrease reported “surprise box” issues and improve the overall customer experience. After testing the email using Optimizely Full Stack, they saw a 37% reduction in unexpected order reports from customers.

Scaling experimentation

With a consistent framework in place, the number of people at Blue Apron now performing testing has expanded well past the Growth/Member Experience team. Anyone in the company on any team can create a test, see the results, analyze it and use that data to inform their decisions.

"Now that we have this capability, other groups have started using it. We went from one or two teams doing one or two tests a quarter to now, when we probably have at least 10 tests live at any given moment, and a large number of tests every quarter being run by every product team." - John Cline