A few years ago, Marks and Spencer, the British retail company, launched a multi-million dollar redesign of their website. The redesign wasn’t just costly, but also took two years to complete! After the redesign was finished, online sales actually decreased by 8%, and a lot of loyal customers felt frustrated by the experience.
So, what happened? Well, two years is a long time, and the way we interact online changed significantly over the course of the project. Not only that, but the complete redesign of the user interface turned off customers. Such a sudden and dramatic change meant that visitors had to completely re-learn how to use the site.
Marks and Spencer also moved away from the Amazon platform, losing all saved data in the process. That meant visitors who had saved passwords and items could no longer access these items or accounts. Imagine how frustrating that must have been!
Could this negative impact on sales and customer service have been avoided? Absolutely! Testing incremental changes over time would have given Marks and Spencer customer data on each change so they could make changes that improve sales and revenue before rolling them out. A simple experiment could have mitigated any risk and helped them make more effective business decisions.
Of course, it’s easy to see that Marks and Spencer made a number of big mistakes. They didn’t iterate their changes and made it hard for existing customers to learn the new site. They took too long to make their changes and fell behind the curve for web design. Their transition away from Amazon was anything but smooth, and it put a huge burden on their customers. And they suffered for it.
Of course, their biggest mistake was not experimenting in the first place. They implemented all these changes with no idea how their customers would respond. If they’d experimented properly, they would have learned which changes would work and which would fail. They’d have a better idea of what was worth keeping and what would just confound customers.
But experimenting means more than just bombarding customers with as many variations as you can come up with. To get accurate results, you have to let an experiment run until it gets to statistical significance. Then you have to review the results and decide how best to proceed from there.