Channel 4 A/B test their way to primetime success using Optimizely

Channel 4 is a premiere British public-service national television broadcaster that first launched in 1982. The channel is known for its distinctive programming, which often pushes boundaries and tackles controversial topics. Channel 4 has been credited with launching several influential and ground-breaking television shows and documentaries.

Commercially funded, Channel 4 is owned by the Channel Four Television Corporation, a public body established by the British government.
Channel 4’s popularity stretches beyond terrestrial television; the TV broadcaster also allows its viewers to watch its programmes online through its free video on-demand service, stream through users’ SMART TV or watch via a dedicated mobile app. 

To improve the customer experience across its multiple channels, encourage users to watch more content and test and validate experiments to maximize advertisement revenue, Channel 4 partnered with Optimizely in 2017.   

Lack of clarity leads to blockbuster change

Internally, Channel 4’s product management and experimentation teams walk a tricky tightrope. The teams must carefully innovate and optimize its on-demand platform and apps to attract and engage users on a budget whilst ensuring that features and changes that are made are released with minimal risk and disruption to both its users and advertising partners.

As Andrew Jackson, Senior Product Manager at Channel 4, detailed, “we wanted the opportunity to be able to try different things cheaply, and ultimately reduce and minimize the risks of launching new features.” 

Before utilizing Optimizely’s Web and Feature Experimentation platforms, Channel 4 could lack clarity when testing new ideas. Whilst small group user testing provided some insight, the internal teams at Channel 4 lacked the ability to truly test and validate assumptions at scale. 

Also, at the time, it was difficult to measure the impact of individual features and differentiate the impact of product changes from schedule changes. Laura Shelton, Optimisation Manager at Channel 4, explained that “as a media company, you can release features and try to measure the impact of them, but usage of our services is influenced by which popular shows are on at any given time.” 

Therefore, Optimizely’s full experimentation suite, containing both Web Experimentation and Feature Experimentation products was selected by Channel 4 to support the TV broadcaster with its A/B testing journey.

After a smooth onboarding process, Channel 4, supported by testing partner Good Growth, quickly started to leverage Optimizely’s experimentation platforms to test various hypotheses.

When conducting tests, Channel 4’s internal teams work in multi-discipline ‘squads’ who each own a particular strategy. To determine the focus and direction of these squads, workshops are held over a two-week sprint period. 

Data and insights from users, customer panels and competitive research are all collected to shape ideas for testing. These insights are used to produce an ‘Experiment Card’, which details the technical build of the experiment. 

The Experiment Card is then used by the delivery teams to build and quality assurance (QA) experiments, with the Experimentation team’s guidance on targeting and segmentation. For experiments utilizing Optimizely’s Feature Experimentation platform, tests usually require a release via an app store before the experiment begins. Otherwise, all tests in general must avoid big content launches.

Channel 4’s web testing partner, Good Growth, align with in-house developers to provide testing support on the website. They work closely with Channel 4’s Experimentation team to ensure tests are run in a smooth and impactful manner.

Channel 4 switches customer experience with Optimizely

Use Case 1 – AutoPlay Experiment

One goal of Channel 4’s on-demand platform is to engage its users and make it as easy as possible to watch content.

Channel 4 hypothesised that reducing the length of time taken for content to autoplay into the next episode would keep viewers watching more content.

The team utilized Optimizely’s Feature Experimentation platform to run a feature variable test that trialled various countdown timer lengths including some that the business was nervous might be too short.

The test was a fantastic success, removing friction between episodes, and delivering surprising results. The optimum countdown timer length between each episode was shorter than expected, proving to be the ideal time to best engage viewers.

Use Case 2 – Picture in Picture Viewing

Picture in Picture (PiP) viewing is a feature that enables users to watch a video in a small floating window while using other applications or browsing the web.

After receiving consistent user reviews and feedback around the lack of PiP viewing, the Channel 4 product team wanted to incorporate the feature into its mobile app. However, there was concern both internally and from stakeholders that advert breaks would be less impactful in a smaller viewport.

To test this, Channel 4 utilized Optimizely’s Feature Experimentation to run a feature flag test to track and monitor when users were minimizing the screen and if these times coincided with ad breaks. The results of the experiment showed that users minimized the screen at consistent intervals whilst viewing content, not just whilst ad breaks were being screened.

By suggesting an experiment to explore the impact of PiP, Channel 4’s product team were able to show that the feature improved customer experience so much that they watched more content and therefore also generated more advertising impressions. By vouching for the customers’ experience, whilst keeping business needs in mind, the team were able to design the experiment and tracking needed to answer questions from both sides.

Channel 4’s product and experimentation teams noted several positive benefits from utilizing Optimizely’s experimentation capabilities. By utilizing Optimizely’s Stats Engine, Channel 4’s internal teams can quickly and efficiently monitor experiments while live. Laura Shelton noted that her team appreciated both the simplicity and accuracy of Optimizely’s platform whilst also having the ability to dig deeper into the results and view convergence over time before experiments reach statistical significance. 

The increased flexibility that Channel 4 now have to run complex experiments, track events and define audiences has not gone unnoticed by those inside the organization. Laura Shelton commented that “because Optimizely’s web testing is so quick, easy and agile, we can at least explore ideas and see if it's then worth taking that forward into the more complicated areas to experiment on.” 

Shelton shared that Optimizely’s platform allowed Channel 4 to launch features that help to isolate changes and assign value to the change that was made. This also allows Channel 4 to test in a more lightweight fashion as the isolated feature can be changed, experimented on, and terminated if the experiment fails.

utilizing Optimizely’s Web and Feature Experimentation products allows Channel 4 to quickly and effectively run both simple and complex experiments on its various platforms. The Events feature in Optimizely’s Web Experimentation product, which tracks the actions that users take on Channel 4’s website, has given the product team flexibility and leeway to run experiments when a variable has been misinterpreted or implemented incorrectly.

From a user management perspective, Channel 4’s internal teams found value from the environment permissions feature. The feature controls which collaborators have permission to start and stop experiments in an environment. This allowed Channel 4’s product team to limit the number of people who can change the statuses of experiments or move them from one environment to another. This proved of particular importance for the development teams on full stack experiments.

Andrew Jackson explained that this feature helped to streamline and understand experiments that were taking place concurrently as, previously, experiments would have to be duplicated across multiple projects.

There has been a culture shift and mindset change inside the organization towards experimentation. Previously, features and releases would be guided by pre/post analysis, sometimes supported with small user group research tasks. Now, there is a wider demand from senior leadership to product and marketing teams that ensures hypotheses are validated and tested to mitigate risk, as well as prove impact on KPIs and find the right direction to launch new features in.

The number of experiments deployed by Channel 4 has grown rapidly with more than 150 A/B tests launched and documented in the past year. All experiments are now documented in a single repository. An experimentation community has been built around this repository, with comments and advice on experiments encouraged. This gives experimenters more feedback and thus opportunities to learn. It also gives others the opportunity to learn at the periphery.

Conclusion
By utilizing Optimizely’s experimentation platform and partnering with Good Growth to scale their A/B testing program, Channel 4 have realized the full potential that implementing an extensive experimentation testing programme can bring to an organization. Now, hypotheses can be tested and ideas can be validated.

Going forward, Channel 4’s product team aims to hone its platforms’ personalization capabilities to create bespoke and tailored experiences for its users.