Driving Innovation at HP through an Experimentation Center of Excellence

Looking at a comparison between Fortune 500 lists in 1955 and 2016, there’s a startling observation: only 12% of companies still remain. The companies that remain on the list are the ones that have invested heavily in R&D, retaining employees for the long-term, and exploring strategies to outpace the market. Organizations that endure have a long-term strategic advantage, strong product-market fit, a vision for the future, and a cohesive team.

It’s this reason why HP is finding creative ways to expand its reach. With 49,000 employees, ideas for experiments are everywhere throughout HP. Staying true to their organization-wide motto to “keep reinventing,” leadership has established a center of excellence for experimentation and customer experience optimization.

HP’s goal is to ensure that teams throughout the company have the independence they need to run experiments that make HP’s operations, products, and end customer experiences better. Teams can be more confident in their decisions using data as the northstar. As a result of their focus on experimentation, they’ve been able to run almost 500 campaigns and have driven an incremental $21 million in revenue with Optimizely. Here’s a look into what the team at HP has learned.

Building a Center of Excellence

HP self-identifies as being in the early stages of their journey towards creating a true culture of experimentation throughout their organization. At the same time, they know that increasing the pace of innovation for their campaigns will be essential to ensuring peak performance for various digital business units that have started doing experimentation, including their B2C online store, HP.com website, B2B e-commerce organization, Instant Ink organization, and Customer Support site.

HP has invested in creating a program around experimentation, building a more robust process for ideation, prioritization, development, execution, and post-campaign analysis. HP also documents their historical campaign performance in a wiki to ensure that everyone who is involved in experimentation can learn from past campaigns. The company shares learnings with business stakeholders throughout the organization in their weekly program reporting meeting, and are also planning to roll out a weekly newsletter to help share updates and findings across their teams.

“Before, we had a less structured A/B testing program,” explains Neville Davey, Digital Experimentation Worldwide Program Lead at HP. “Our team is built around a long-term vision and goal to improve the customer experience and financial performance for all digital business units within HP.”

Ten team members at HP are responsible for building the experimentation program as a significant portion of their responsibilities. The team includes three program managers, five developers, a UX designer, and an analyst. Davey oversees the program and the collaboration between these team members, who support business users across HP.

The long-term goal for the experimentation program is to increase the volume and velocity of their program. The center of excellence is tasked with driving a company-wide data-driven digital mindset within HP. Quarterly executive reviews and planning, a resource center to enable other business units to kickstart optimization programs, and the sharing of learnings are evangelizing an optimization mindset.

Anchoring the Center of Excellence to Rigorous Growth Milestones

HP has been paying close attention to the relationship between the number of active campaigns, the number of visitors impacted, and the attributed revenue contribution. Since the program first launched in 2015, there have been important improvements in key business metrics, including revenue.

“My first priority has been to create a report and test plan templates in PowerPoint,” says Anmeen Leong, Digital Experimentation Program Manager for the Americas. “These templates are standardized and clean enough to help business owners showcase key tests and milestones to senior management.”

This reporting process ensures that every A/B test ties back to a long-term business goal within HP. The company has not only built a structure but established clear foundations to make data-driven perspectives available to anyone running an experiment.

“To quantify the benefits and results of the program, we’ve put more time into reviewing the way we do analytics on each test,” says Leong. “We’ve recently rolled out a more robust process for ideation and prioritization to help us align on the most impactful tests, and to increase the velocity and value of our testing program.”

Planning to Scale Sustainably

HP’s ultimate goal is to give every employee the chance to experiment and gather data around their ideas. The center of excellence is exploring ways to help scale HP’s testing program sustainably. As a result of their focus on experimentation, they’ve been able to run almost 500 campaigns and have driven an incremental $21 million in revenue with Optimizely.

The goal is to democratize testing but also centralize core business initiatives. A top priority for Leong and his team is to help teams build testing into their workflows—to enable these teams to use data to answer critical business questions. Here are some example tests that HP’s Center of Excellence has overseen:

• HP’s omnichannel team wanted to see if shoppers were more responsive to certain resellers vs. others. They added Best Buy, Amazon, and Staples links and then tracked orders to see which were driving the most online sales. This data helped HP figure out how to most effectively work with partners.
• HP’s Instant Ink subscription service experimented with different enrollment offers. The experiment involved showing customers an option for a free trial and positioning the service as a printer feature. The experiment increased enrollment by 37%.
• The eCommerce UX team redesigned its printer category page and conducted a 50/50 split test of the old page vs. the new one. The new page increased success metrics by 20% and yielded an incremental $260K per quarter.

The center of excellence uses early wins such as these to evangelize experimentation across their entire organization. “With these instant, early results, we’re planning to scale the use of Optimizely company-wide,” says Davey.