Testing in production

What is testing in production?

Testing in production (TIP) is a software development practice in which new code changes are tested on live user traffic rather than in a staging environment. It is one of the testing practices found in continuous delivery.

Production software is the version of software that is released live to real users. In contrast, development, staging or pre-production software is in the process of being built and is not yet available to end users.

Why test in production

Historically, companies have tried to ensure that the software they build has been thoroughly tested for bugs in development, staging and pre-production environments, well before it reaches users in production. Catching bugs early prevents users from seeing errors, increasing customer trust and overall satisfaction with a brand and its products.

However, catching all bugs in development and staging is not easy. Engineering and QA teams can spend a lot of time and effort building unit tests, test suites and test automation systems, trying to simulate the production environment, or manually verifying user flows with mock user data and test cases to try to expose bugs only to find out that a key corner-case was overlooked. In the end, many users may experience buggy software even after a large amount of time is spent testing in development.

In many instances, it is impossible to completely simulate live, real-world software in a test environment. With all the dependancies present in modern production systems and the many possible edge cases, production testing has become a necessary part of devops and software testing. Top software companies such as Google, Netflix and Amazon constantly release new features to a fraction of their traffic to measure the impact.

Production testing and feature flags

With the advance of feature flags (a.k.a. feature toggles and feature rollouts) that allow engineering teams to expose new software to only fractions of live production traffic, companies can put experimental or new features in front of a small portion of their production traffic to quickly verify their software works as expected in real-time while having a safe way to roll back any uncaught bugs using a feature flag kill switch or rollback.

Running tests in production via rollouts or feature flags allows for all the product data, dependencies and edge cases to be accounted for in comprehensive integration tests. Having real-world data can be especially powerful when doing performance testing or load testing.

Feature flag tooling also has the added benefit of allowing for A/B testing, where the new feature is compared against the previous version of the software to see which one results in a better user experience based on production data. This allows software engineers to not just ensure that their new features are bug free, but also uses real data to validate that the change actually improves their overall software experience.

If your team is looking to do tests in production, Optimizely Rollouts offers unlimited free feature flags and rollouts built on an enterprise-grade platform.