Test and learn to move fast and fearlessly

How to move fast and fearlessly

Optimizely VP of Product, Jon Noronha, kicked off the day discussing how embracing techniques like feature flagging, progressive rollouts, and experimentation allow teams to measure impact continuously. Taking a Test and Learn approach to development and delivery ensures you can understand how ideas and product changes impact your customers. 

Jon announced two key features from our paid Feature Experimentation plans – experimentation and feature configuration – are now available in our free Rollouts plan for Feature Experimentation. Anyone can sign up for a free account to bring Progressive Delivery and Experimentation practices to their product development process. Now everyone can use data to deliver better products.

Highlights from the event

In the Test and Learn keynote, Nicole Forsgren, author, DevOps expert and VP of Strategy and Research at Github, talked about the roles of process, technology, and culture in delivering essential software. Forsgren asserted that innovating to go faster than your competition is really about testing new ways of doing things. There is no fully automated, out of the box solution that will give you a competitive advantage, the best performing teams align strong technology solutions, processes, and culture to deliver higher quality software, faster. She also highlighted the importance of avoiding heavy-weight change management practices, which can actually slow you down and introduce instability. Instead, she recommended automating low-risk approvals. Forsgren explained that to maintain stability, and ensure you are building the right products, you can automate a lot of testing. But certain testing must remain manual, and this doesn’t have to slow down how frequently you integrate and deploy code. Progressive Delivery decouples these processes. Adopting feature flags are the foundation for Progressive Delivery, enabling A/B testing, blue-green deployments, canary testing, gradual rollouts, and safe rollbacks. This is how to empower teams to take code ownership, starting with developers. 

Ellen Chisa, CEO, and founder of Dark, takes the concept of testing in production a step further in her talk Programming in Production for the Fasted Feedback Loop. She demonstrated how to create an infrastructure focused on getting code into production immediately, allowing developers to build and iterate faster using real user data as you go. For many teams, testing and coding in production environments are counterintuitive and stressful concepts. Asa Schachar and Justina Nguyen from Optimizely’s Developer Relations team addressed how to instill practices that remove the risk of causing the ultimate fear where an action you take could bring an application down in production. In Deploying Fearlessly in a Continuous World, they explain how adopting feature flag driven development gives product development teams full control over rolling out and rolling back in production for safe, testing, iteration, and experimentation independent of code deployments.

Product leaders also spoke about experimentation strategies for moving fast and fearlessly. For example, Luis Trindade, Senior Product Owner at luxury e-commerce platform Farfetch, encourages teams to celebrate failures when new products and A/B tests yield unfavorable results. Experimental features that disprove a hypothesis are an opportunity to learn and inform the next thing you prioritize and build. Losing tests should be discussed as they are likely to provide the best insights for improving your product. Oji Uduzue, VP of Product at Calendly, discussed how to Run High-Impact Experiments with High-Quality Customer Discovery. He spoke about being fearless in your Test and Learn approach, taking “big swings” with your product experiments to make an impact and ensure that your product fulfills your user’s needs, motivations, and optimal workflows. 

Aleksandra Vangelova and Abdelrahman Hussein from the Product and Engineering teams at the international meal kit company, HelloFresh explained how they developed an architecture for retention experiments on top of Optimizely. They shared how to measure the impact new features have on retention metrics like revenue per customer, average order value,

and other indicators for how much customers are spending and at what frequency. Gauging the down-funnel effects of new features takes more time than monitoring short-term conversions. Since they launch 99% of new features as experiments, the duo needed to ensure testing was not slowing down their software development cycles. The team needed to architect a sustainable way to test many new features at once. In the talk, Vangelova and Hussein shared their “sticky variation” concept, which allows them to run more retention experiments across their web, iOS, and Android applications using a unified profile, and microservice approach. Their experimentation architecture enables them to deliver products faster and more confidently.