Canary testing

What is canary testing?

Canary Testing is a way to reduce risk and validate new software by releasing software to a small percentage of users. With canary testing, you can deliver to certain groups of users at a time. Also referred to as canary deployments, incremental, staged, or phased rollouts, canary releases are a best practice in devops and software development.

The origin of the phrase is from the phrase “Canary in the coal mine”, in which coal miners would bring a caged canary bird into the coal mine to detect if the level of toxic gas was too high. Since birds have a lower tolerance for toxic gases than humans, if the bird died it was an early warning for the miners to evacuate. In the context of software development, the outcome isn’t as gruesome, instead as software developers the goal is to quickly detect and resolve an issue with new software before it degrades everyone’s experience.

How to do canary testing

Canary tests can be done using blue-green deployments to split traffic at the server-level to slowly roll over traffic from one version of an application to a newer version of an application using a traffic router at the server-level.

canary testing blue-green deployment Splitting traffic by application version

Or can be done more granularly at the feature level using feature flags to send a percentage of end users to the new version of the experience.

canary testing feature flags Sending 1% percentage of users to new experience using Feature Flag

How to do canary testing

Canary testing allows new code or features to be released to a small subset of users to verify if there are any issues with the code before releasing to a larger audience. By limiting the release to a select audience to minimize the blast radius of new releases, teams are able to validate functionality and performance prior to rolling out to all users.

Another reason to do canary tests is that often times development or staging environments do not fully match production environments and by testing with a small percentage of production users (often referred to as testing in production), you can detect issues that may not have been found in staging or development environments.

Canary testing & feature flags

Canary testing can be done by using feature flags which enable teams to separate code release from feature enablement and turn features on and off remotely for specific groups, percentages of users, or all users. Using feature flags, teams can limit the release to just 1% of users, monitor key metrics like error rates, latency, and business metrics to ensure the new feature has no negative effect.

If the canary test discovers an issue during the deployment process, it’s easy to disable that new feature or code by turning that feature flag off. Canary releases can help prevent major downtimes, lost revenue, or negative customer sentiment by providing fast learnings about the performance of a new feature while limiting those affected.

Canary testing & continuous delivery

Championed by influential software developer Martin Fowler, continuous delivery is a software development discipline in which software can be released to production at any time. It has been adopted by leading software companies around the world whose teams are now expected to quickly deliver software that is stable and bug-free to your user base.

Canary testing fits into the continuous delivery and continuous integration methodologies. Best in breed continuous delivery systems like a companies like Facebook integrate deep monitoring and automated testing into their canary testing. If a feature fails a monitoring check once it’s been deployed as a canary to a percentage of users, it will automatically roll back. With continuous deployment and canary tests, a team can more safely release new functionality and code changes at scale.

Canary testing in Intelligence Cloud

Software engineering teams can now use Optimizely free feature flags to implement feature flags for canary testing.

Teams can embrace the practices of continuous integration and continuous delivery employed by today’s advanced engineering teams. Both teams – and entire companies – will be able to benefit from the faster development cycles, and feature management practices like canary testing to mitigate risk.

Optimizely server-side experimentation customers can follow in the footsteps of the world’s top software companies (like Amazon, Google and Facebook) that validate their features to just a small percentage of users before rolling out more broadly.

Optimizely also offers enterprise-grade feature management and A/B testing through Optimizely server-side experimentation, a powerful solution for doing canary testing, controlled feature releases, and feature tests in your product or app.