Feature test

What are feature tests?

Feature testing is the software development process of testing multiple variations of a feature to determine the best user experience.

With feature tests, you can validate whether a new feature for web page or app is a good fit. At the same time, you can also determine what variation of the feature will be most successful. If the feature test finishes successfully, you can then rollout the feature with the configuration of the winning variation of the feature test.

How feature tests work

Feature tests are similar to A/B/n tests that allow you to control whether for each variation the associated feature is on or off via feature flags (aka feature toggles). It also allows you to control the feature variable values for the various variables associated with the feature.

As visitors interact with your website or application, the feature test accumulates metrics related to their engagement with the feature. You can then look up statistics on the analytics dashboard to determine whether the feature leads to positive or negative behavior. At the same time, you can also determine what configuration of the feature is the more successful.

The benefits of feature tests

There are a number of use cases for feature testing:

Validate feature fit

Feature tests allow you to accumulate metrics and validate whether a feature is a good fit and satisfies your business requirements.

By setting up a feature test, you can enable the feature for a certain portion of the traffic and disable it for the rest. This way you can measure how your metrics are performing when the feature is on and compare it against when it is off thereby validating if the feature will be good to eventually rollout the new functionality to everyone using the product.

Determine the best feature configuration

In addition to enabling / disabling your feature at the variation level, one can also run a feature test to experiment on different configurations and test scenarios for the feature. That way it will be possible to determine the best feature experience.

Discover bugs

In addition to standard software testing practices such as unit testing and functional testing, feature testing can be a good way to discover bugs and issues that may only arise in edge cases of your production environment. By wrapping new features in feature flags, you can quickly rollback any bugs without a code deploy.

Iterate to perfection

For long term features, you can run feature tests on them again and again and determine the best configuration for them over their lifetime thereby leading to the best experience possible.

Feature tests and continuous delivery

Introducing feature tests into your feature delivery process allows you to quickly validate ideas, look at metrics and determine if the feature is ready to be rolled out to everyone all without requiring you to do another code deployment. You can also tweak various aspects of the feature by experimenting with different configurations.

Once you are done collecting metrics and happy with the results, you can then roll-out the feature gradually to everyone again without the need to deploy code.

Optimizely and feature tests

With Optimizely’s server-side feature management capabilities, product and engineering teams can instrument feature flags and run feature tests on them. This allows them to validate their hypotheses quickly, measure impact of the feature and then gradually roll the feature out. All this can be accomplished by instrumenting the feature flag once and then using Optimizely to experiment on it and roll it out.

In Optimizely when evaluating a feature, the related feature test is evaluated first to determine if the user can be granted access to the feature through the test. If not, the related rollout is evaluated to determine feature access.

This works great for product development teams since risks can be mitigated early by running feature tests before a lot of effort goes into building and maintaining the feature. It also gels well with best practices of continuous integration and continuous delivery since in Optimizely feature tests and feature rollouts are closely coupled and once the test achieves the desired results, one can rollout the feature without the need to make any changes to code and re-deploy.

Optimizely currently offers free feature flags through Optimizely free feature flagging, a completely free solution for controlled feature releases in your product or app. Rollouts offers a number of SDKs on GitHub for some of the most popular programming languages and frameworks including Android, C#, Go, Java, JavaScript, PHP, Python, React, Ruby and Swift.

Feature tests are currently available as part of Optimizely Full Stack. You can check out the plan most suitable for your needs here. Again, you can always start with Optimizely Rollouts and upgrade your subscription to unravel the full suite of features.