E-“quip”-ping product teams to be more data driven

Sometimes tracking a key metric to validate an experiment is only half the battle. This sentiment is shared by the product team at quip, an eCommerce store where you can buy next-generation electric toothbrushes and other oral health supplies. Recently, they learned that making sure your experimentation platform, Optimizely in their case, integrates with the other tools in your stack, is key for producing reliable data.

Using Optimizely Stats Engine and integrating with Segment, a customer data platform, quip built an experiment to track whether a new digital shopping experience would lead to increased product bundles for each subscription sold. As quip Sr. Digital Product Manager Katie Furr Amos told us in this webinar, the team was able to confidently point to a 6 percent boost in key subscriptions. Here’s how they pulled it off.     

Experimenting against the status quo

quip, the makers of subscription-based oral health products, hit an inflection point with the lifeblood of the business—its ecommerce platform. Due to high growth, the digital shopping experience needed a paradigm shift to support the number of new products and bundles quip now offers. For the company to achieve its next stage of growth, the product team sought a redesigned shopping platform that enabled long-term revenue goals. 

The team ran three experiments using Optimizely to test different iterations of the ecommerce experience that offered shoppers various combinations of product bundles, types of subscriptions, and buy-once experiences.

The shopping experience for quip products and bundles

The baseline expectation for the redesigned page was that it produced as many key sales—the number of products bundled together per subscription—as the status quo. The team would determine the success or failure of the experiments by tracking this primary metric. However, evaluating the effectiveness of the new design entailed additional analysis around metrics such as the number of toothbrushes per order, the ratio of different colors sold, as well as conversion rates, average order value, and subscription attach rate. 

There was just one hitch. 

Ensuring apples-to-apples comparison across platforms

While the team had great specificity around which primary metric to track, the logic behind measurement differed between Optimizely and Looker, quip’s business intelligence platform. There was a slight difference in the way each solution defined the number of products bundled together per subscription. And the discrepancy undermined early experimentation results.

By uniting data from Optimizely and Looker via Segment, the team was able to instrument a consistent set of events that established a single source of truth for data to be shared across tools. This way they could effectively experiment, and by the third iteration of design, the team had achieved their goal. In the winning design, quip tracked a 6.1 percent increase in product bundles for each subscription sold, and an 8.3 percent jump in subscription attach rate. 

The product team also made gains beyond just a successful experiment. By having Optimizely at their fingertips, quip was able to initiate new ways of approaching metrics, business decision making, and product exploration. The exercise laid the groundwork for a culture of experimentation and a trusted process to implement big product changes.

Ready to drive experimentation to build amazing customer experiences?

With Optimizely, quip found a reliable experimentation platform that integrated with third-party tools and fueled intelligent decision making. If you’re a marketing product manager in search of an experimentation tool, Optimizely can help you build surefire customer experiences. 

Get the full story from quip here.