Making Headlines with Experimentation

Get Started

One publisher continues to make headlines by overcoming challenges with experimentation.

After a century and a half of reporting the news that matters most to Minnesota, the Star Tribune looked to extend its digital reach in an all-too-competitive marketplace. By embracing Optimizely and its robust experimentation platform, Star Tribune grew its digital subscriber base and found a better balance of consumer and advertising revenue.

35%

Increase in digital subscription volume year over year

20%

Lift in overall revenue from free article experiment

37%

Lift in subscription conversion rate from free article experiment

Products Used

Optimizely Web X Personalization

Deliver tailored experiences across every customer touchpoint

Learn More
Star Tribune

Learn More
“Optimizely helped us understand what works, and what doesn't. So we now do more of what works, and less of what doesn't, while impacting business in a more sophisticated, more ambitious way.”
Patrick Johnston, Director of Digital Product and Analytics

A New Era In Publishing

The Star Tribune recently turned 150 years old. There are a number of reasons the company has thrived this long, especially during these trying times for news publishers. With 250 journalists, multiple Pulitzer Prizes, and national acclaim, you would expect great things. But a big part of its success can be attributed to the company’s online division, its heavily trafficked flagship site, and its Director of Digital Product and Analytics, Patrick Johnston.

As with many media companies, Star Tribune has seen a sustained decline in print revenues since 2005 and Like most publishers, StarTribune realized it needed to offset the losses by increasing its online revenue. When the 2008 recession hit, and advertisers began tightening their belts , it was clear that digital subscriptions would be crucial to its continued success.

Going Digital Wasn't Enough

Star Tribune was the first major-metropolitan news organization after the New York Times to launch a digital subscription product. And while the initial results looked promising, sales slowed in 2015 despite efforts to redesign their site and improve their digital channel marketing. Business was plateauing.

Just a year earlier, Johnston had joined the team. He noticed that there was ample data to assist decision making, but no experimentation methodology in place to help understand the data’s practical impact on business functions. It was clear that the organization could benefit from a holistic way to measure the incremental impacts of various changes they were making to their digital journalism and advertising products, particularly when factoring in seasonal swings and the constantly changing news cycle.

Perhaps the most pressing challenge they faced was how they could best optimize site revenue. Their metered paywall approach allowed them to keep site traffic robust in order to sell ample advertising. But by asking more visitors to become subscribers, they could increase circulation revenue. Somewhere in between was the answer.

Testing the Hypothesis

Johnston’s belief was that StarTribune.com could achieve optimal revenue by lowering the number of free articles that viewers could access before being asked to subscribe. Increasing subscription revenue, he hypothesized, could offset any decrease in engagement that affected ad revenue.

Using Optimizely X Web Personalization, Johnston’s team experimented with three test variations. The first and current offer included 10 free articles before readers were required to subscribe. The second permitted access to 7 free articles. The final version offered 5 free articles.

They assumed the 5-article experiment would drive the highest subscription rate because it interrupted readers earlier. What they discovered surprised them.

Exceeding Expectations

5-article variation outperformed 10-article with a 49% lift in subscription revenue, and a -12% decrease in ad revenue, leading to a +15% lift in overall revenue. However, the 7-article offer had a +37% subscription revenue lift, but half of the decrease in ad revenue (-6%), leading to +20% lift in overall revenue against the 10-article variation.

Without experimenting, Johnston admits, they likely would have proceeded with the 5-article offer and taken the hit on ad revenue, missing the opportunity to learn what actually works best. Being able to account for changes in both subscription rates and ad engagement also renewed the team’s interest in conducting additional tests on functions such as personalized article recommendations and newsletter acquisition in order to drive customer engagement. 
Ultimately, the team was able to increase subscriptions by 35% year over year.

A Total Transformation

Johnston now believes that experimentation has completely transformed his organization. “We've gone from sitting around a table debating hunches and opinions,” he confesses, “to trying things out on a small percentage of our traffic with open minds and eager anticipation.”

He also has this message for those just dipping their toes into the experimentation world—don’t be timid. “The idea of subjecting your site to a new platform, experimentation, and personalization can all be pretty intimidating,” Johnston points out, noting that, “Once you start, you realize it's not so scary and it actually mitigates the risk of making changes.”

Perhaps the most compelling observation Johnston made was this: experimentation changes people. They begin to think differently, feel more involved, see the impact of their ideas, and with that comes a tremendous amount of satisfaction.

Just Getting Started

With site revenue up, Johnston’s team now wants to focus on nurturing customer relationships. Naturally, they plan to improve the way they move non-subscribers through the conversion funnel, but they also want to understand customer preferences and be able to personalize each customer’s experience.

“I firmly believe that gaining the ability to quickly make changes, see the impacts, and then move on to the next idea was essential to our success this past year." — Patrick Johnston

Thanks to Optimizely, Johnston is confident they can add value at each stage of their journey.

Get Started with Optimizely

It's your turn. How will you experiment everywhere?

Get Started