Blue Apron Validates Products Faster with Experimentation
Blue Apron Embraces Experimentation To Deliver Better Experiences
Blue Apron creates incredible experiences. Founded in 2012, Blue Apron is a consumer lifestyle brand that symbolizes a love of home cooking, excitement and discovery, and deep, meaningful human connections.
Blue Apron’s core product is a meal experience that customers create with the original recipes and fresh, seasonal ingredients that are included in every box. All recipes are accompanied by printed and digital content, including how‑to instructions and stories of suppliers and specialty ingredients. Blue Apron also sells wine, which can be paired with its meals, as well as a curated selection of cooking tools, utensils, and pantry items, which are tested and recommended by Blue Apron’s culinary team.
Blue Apron has developed an integrated ecosystem that employs technology and expertise across many disciplines. The company’s supply-demand coordination activities––demand planning, recipe creation, recipe merchandising, fulfillment operations, and marketing––drive its end-to-end value chain.
Blue Apron uses Optimizely X Web, Full Stack, and Mobile to experiment across their customer lifecycle to validate and prove the impact of new or redesigned features.
10x
Increase in experiments run per month after adopting Full Stack
100%
Number of product teams using experimentation
37%
Decrease in customer issues related to unexpected orders
Products Used
Optimizely X Full Stack
Empower your product development teams to experiment everywhere.
Learn MoreOptimizely X Web
Enable everyone in your organization to improve every experience.
Learn MoreMore from Blue Apron:
We can now get an early signal from the Optimizely Results Dashboard within a few days, whereas before it would have taken weeks or longer for someone on our analytics team to pull the data and understand what was going on.
John ClineEngineering lead, Growth/Member Experience
Getting Started with Experimentation
Since their inception, Blue Apron has been a data-driven, customer-focused organization, and because the category of online food is growing and constantly evolving, experimentation is in the company’s DNA.
In the early days, Blue Apron used Optimizely Web for testing. It was easy for non-technical users to create and launch experiments without engineering support, and the tool also worked well with their single-page application. But there were some drawbacks. Optimizely Web only worked with client side changes on the web, which limited Blue Apron’s ability to run experiments to optimize their mobile apps or back end applications.
Going Beyond Conversion Rate Optimization
At Blue Apron, a large part of the business is powered by scheduled back end jobs. Order charging, turning digital orders into physical packages, creating shipping labels, choosing recipes for users and sending emails all happen offline. A client-side solution provided limited flexibility when it came to testing these experiences.
Also, running experiments and measuring key KPIs required work from the analytics team, and turnaround time could take anywhere from two to four weeks. This greatly impeded Blue Apron’s ability to act upon needed changes quickly. So while they continued to use Optimizely Web, they also required a solution specific to their server side use case.
John Cline, Growth/Member Experience Engineering Lead stepped up to identify a solution that would allow Blue Apron to test at more stages in their funnel.
Deciding With Confidence
Just when John and team were weighing options and solutions, Optimizely released their new offering, Optimizely Full Stack. The solution targeted Blue Apron’s precise experimentation use case and provided mobile and server-side SDKs for testing within their code base.
With Optimizely Full Stack, Blue Apron gets statistically valid experimental results on their server and mobile applications which helps them make faster, more informed business decisions. With Optimizely’s real-time results, Blue Apron was able to avoid launching a redesigned reactivation flow, which would have resulted in a significant revenue loss.
Blue Apron can now quickly and easily build and set up new tests that were simply not possible before.
Reminding New Users of Upcoming Deliveries: A Hypothesis
Because of Blue Apron’s unique sourcing and fulfillment model, customers must decide whether or not to receive an upcoming order shortly after their previous box arrives. While this system works well for experienced customers, new users often require more hand-holding. They noticed that new customers were more likely to miss their order management cutoff point in their second week. Blue Apron hypothesized that customers that received these “surprise boxes” had poorer retention dynamics, and consequently poorer lifetime values than those who avoided surprise boxes through proper subscription management.
They tested reminding new users that their cutoff was approaching, and encouraging them to manage their upcoming deliveries to see if it would decrease reported “surprise box” issues and improve the overall customer experience. After testing the email using Optimizely Full Stack, they saw a 37% reduction in unexpected order reports from customers.
Scaling Experimentation
With a consistent framework in place, the number of people at Blue Apron now performing testing has expanded well past the Growth/Member Experience team. Anyone in the company on any team can create a test, see the results, analyze it and use that data to inform their decisions.
"Now that we have this capability, other groups have started using it. We went from one or two teams doing one or two tests a quarter to now, when we probably have at least 10 tests live at any given moment, and a large number of tests every quarter being run by every product team." - John Cline