Achieving product launch greatness: Beta vs. A/B testing

"..Will the new design resonate with customers?.."

"..Will the experience be improved or not?.."

"..How will conversions be impacted?.."

"..What if it doesnt work?.."

"..Management approved it with some Hippo requirements I didnt agree with - how will their changes affect the success of the release?.."

And so on, and so on...

At least once a week, I am surprised by another product release that risks millions of dollars in conversions, relied on gut feel (rather than rational data) to design and doesnt have feature management baked-in for quick switching if there is an issue.

I also see a lot of releases that don't leverage experimentation for testing the new features. No wonder product launches are emotional.

But it doesnt need to be like this, and you dont need to 'boil the ocean' to improve the process.

Here are my top tips for avoiding sleepless nights with your next release:

First, qualify the nature of the release:

• The risk of making a change. How likely is it that the change you are proposing will have a negative impact on your users or customers? If the risk is high, you may want to consider running an A/B test first to minimize the potential impact.
• The complexity of the change. How difficult is it to implement the change you are proposing? If the change is complex, you may want to consider doing a beta release first to get feedback from a small group of users before rolling it out to everyone.
• The urgency of the change. How important is it to make the change quickly? If the change is urgent, you may want to skip the A/B test or beta release and roll it out to everyone immediately, with feature management embedded for easy no-code rollback.

Next, choose an A/B test or beta release:

• Run an A/B test if:
• You are unsure of how users will react to the change.
• The change is complex and could have unintended consequences.
• You are making a change to a core feature of your product or service.

• Run a beta release if:
• You are making a major change to your product or service.
• The change is new and untested.
• You need feedback from users before rolling out the change to everyone.

And finally, should you consider building the release into a feature flag?

Feature flags help you mitigate risk with controls for releasing in phases - this could be the best way to run a beta, or you might want to release to a certain audience or geo-segment. If there is a major concern, a bug or user pushback, you simply flick the switch to instantly roll back to the previous state.

Everything we have talked about here - A/B testing, Beta release and Feature Flagging is possible with our free feature flagging tool. Plus get all the support you need from our developer resources and slack community.

#BBD0E0 »