VeggieTales is an animated children’s TV series, hugely popular with pre-schoolers in the United States. Since 1993, they have sold over 70 million videos, 15 million books, 7 million CDs, and all sorts of accessories touting the iconic vegetable characters.
VeggieTales wanted to systematically optimize the design of their entire e-commerce site, store.veggietales.com, so they got in touch with my optimization strategy team at Blue Acorn for direction and results.
Their initial research initated experiments with subtle design changes that, after reaching statistical significance through A/B testing, now receive 100% of the traffic as permanent design changes. All together, these subtle changes led to significant changes in revenue and growth:
By first examining their analytics, the Blue Acorn team was able to determine where to start testing and what to test.
At a glance, here are the results they produced for the VeggieTales store:
Let’s look at the hypotheses and variations that produced these results.
The truest metric for measuring the success of optimization testing for an e-commerce site is Revenue Per Visitor (RPV). Increasing conversion rates is easy if heavy discounting is employed, but that would kill company revenue. Constantly attempting to improving conversion rate is still a smart idea. However, e-commerce merchants should focus on RPV their primary Key Performance Indicator (KPI).
Together, Blue Acorn and Veggie Tales set out to iteratively test many locations in their website conversion funnel. Here are some of the successful experiments they ran across the site's homepage, category pages, product pages, checkout flow, and mobile site:
VeggieTales’ category pages had featured very large, bright, distracting banners. Customers landing on the page didn’t have general guidance on where to go, and they were getting distracted from finding products they wanted and continuing to the product page.
Hypothesis: Dropping the banners would result in a higher RPV.
Results: Without the banner, VeggieTales’ category pages produced a 17.4% increase in RPV.
Statistical Confidence Level: 95.3%
Using custom heat mapping tools, the Blue Acorn team confirmed that VeggieTales’ current product pages had very poor spacing, distracting attributes, and lacked organization. They weren’t awful by any means, but they weren’t optimal either.
Hypothesis: Improving the spacing and increasing the prominence of the primary call-to-action will result in a higher RPV.
Results: The product page variation produced a 13.9% increase in RPV.
Statistical Confidence Level: 96.1%
This is where eCommerce conversions happen. However, VeggieTales’ checkout page included all of the same header and footer elements that were found on every other page of the site. They’re good elements, but they serve as unnecessary distractions to a visitor trying to convert to a customer, and can be eliminated using a website optimization tool like Optimizely.
Hypothesis: Simplifying the checkout by removing all of the distractions in the header and footer – except those designed to act as confidence builders—will result in a higher RPV.
Results: The more streamlined checkout page variation produced a 14.3% increase in RPV.
Statistical Confidence Level: 98.1%
The original homepage had a strong callout for free shipping, which is never a bad idea for converting visitors, but it overshadowed the site’s social icons.
However, we didn’t just want to move the social icons to the header, we wanted visitors to be aware of the immense size of the VeggieTales community, which serves as a confidence builder.
Hypothesis: By adding social icons to the header and making visitors aware of the size of the VeggieTales community, visitors will be more likely to have social interactions within the community and consumer confidence will increase.
Results: Not only has VeggieTales gained more fans and followers, but the variation homepage produced an astounding 36.8% increase in RPV.
Statistical Confidence Level: 95.7%
There’s actually nothing mobile about this UI. It is simply the desktop UI viewed on a mobile device. It’s nice in that it doesn’t rely on Flash, which is still not supported on iOS, but it requires a tremendous amount of effort to navigate the site on a smartphone.
Hypothesis: Sending mobile traffic to a mobile-themed UI will increase the mobile conversion rate and revenue per visitor.
Results: Thanks to testing, VeggieTales now knows they can be generating a lot more revenue from their mobile visitors, as this quick, non-customized, mobile-themed UI saw a 28.1% increase in revenue per visitor.
Statistical Confidence Level: 99.1%