Meetup: Building a culture of experimentation at Visa, Strava, and Tripping.com

Teams at these companies have implemented world-class technology platforms to enable every engineer, product manager, and data scientist to make decisions fueled by experimentation and backed by rigorous statistical analysis. But making experimentation successful is as much about culture as it is about technology. The top companies invest in people and process to ensure that cultural change takes root, and that data-driven decision making becomes the norm.

Getting to this point is a journey, and one filled with victories, setbacks, and challenges to overcome. On Tuesday, Optimizely hosted an A/B Testing Meetup and held an experimentation panel discussion with product managers, engineers, and data scientists from companies at various stages in their journey towards embracing a culture of experimentation.

“How can you bring experimentation to a company that has not had that as part of its corporate DNA from inception?” said Steve Urban, engineering manager at Netflix, who moderated the panel. Ramkumar Ravichandran, Director of Analytics and A/B testing at Visa, shared stories of how the financial services giant is adopting a test and learn culture, while Evelyn Cordner, Engineering Lead for Growth at Strava and Tyler Swartz, Director of Product at Tripping.com, provided perspectives from growing companies that have developed high-velocity experimentation programs with much smaller teams.

They shared their wisdom and war stories, and the key takeaways are captured below:

How to Effectively Evangelize an Experimentation Culture

Getting started with experimentation can be tough, even in a smaller company where everyone believes that testing is essential to building a great product. Getting going requires tenacity, buy-in at all levels, and a solid communication plan. Here’s how Ram, Evelyn, and Tyler tackled getting started.  

• Ramkumar Ravichandran (Visa): Recruit an executive sponsor. You need someone who can provide support and keep the program alive, even when tests fail and you face internal resistance.
• Evelyn Cordner (Strava): Sing the praises of the program. And sing your failures. Communicating about failure is as important as talking about success. “When a test succeeds, people expect that, and they don’t always see why you would want to test instead of just rolling something out,” said Evelyn. But when you talk about tests that don’t work, people begin to realize that there’s a huge benefit in preventing revenue loss.
• Tyler Swartz (Tripping.com): Make testing fun. When Tripping.com began their experimentation program, they notified the entire company of every test, and people placed bets on which variation they thought would win. The CEO of the company would occasionally even take particularly good guessers out for a meal.

How to Get Engineers on Board

One challenge many teams face when getting started with experimentation is overcoming objections from engineering that running tests will slow down the team’s velocity or distract from building new features. Some teams question the cost of developing experiences that ultimately may not be deployed to production. How do our panelists get buy in from engineering?

• R.R (Visa): While Tripping.com and Strava adopted an experimentation mindset early on in their companies’ journeys, Visa has brought test and learn culture to an existing engineering culture. For Ram, getting engineering on board with testing has meant helping engineers overcome the challenges of a tight release calendar and cumbersome bug fixing process. Ram’s team uses experiments to deploy hotfixes, which generates goodwill with engineers and more support for running tests.
• E.C. (Strava): At Strava, the engineering team started from a place of wanting to test, since engineers there own the user experience and are responsible for helping achieve business outcomes. For Evelyn’s team, making things easy and lowering the cost of each experiment is the key to driving continued adoption across all of engineering.
• T.S. (Tripping.com): Before Tripping.com began experimenting in their product, the team would often release code or features to production, see a change in key metrics, and then need to roll back quickly. They would try to determine which changes had caused the metrics to move by rolling back changes one at a time. Now that Tyler’s team rolls out every new feature as an experiment, they can know in advance what changes their metrics. “We now avoid rollbacks, and that’s helped our engineers buy in,” he said.

How to Keep Up the Momentum

Our panelists’ advice for long-term success? Keep at it, be patient, and dedicate the resources you need to be successful:

• R.R. (Visa): Invest in the program for long-term success. After getting over initial concerns that the testing team would be a cost center, Visa now has a dedicated squad for testing, with analysts, technologists, and designers, and “test and learn is standard practice, whether it’s on the roadmap or not.”
• E.C. (Strava): Embracing the culture change and ability to be more agile has been key for Strava’s success. Instead of planning for 2-3 big app releases each year, the team now plans in smaller cycles and launches new versions of the app every 2 weeks.
• T.S. (Tripping.com): Keep going after big wins. Tripping.com’s product team now works on building testing into the product roadmap. They look ahead at what they can test, validate ideas before investing in fully building out a feature, and empower all product managers and engineers to build their own tests.

Watch the full recording below: