How to Wisely A/B Test WordPress Headlines With Optimizely

Because of the flexible framework that WordPress provides, sites of all different sizes and industries use WordPress. Sites such as Time, Wired, Bloomberg, CNN and TechCrunch, all use the same free and open source CMS anyone can download and start their own blog in just a few hours.

One of the best features of WordPress is the ability for third-party developers to create plug-ins that help enhance the features that WordPress provides out of the box. With plug-ins, features and functionality that used to take developer’s weeks or even  months to complete, can be accomplished with a few clicks.

Take A/B testing headlines for example. Until recently, only the largest media brands on the web could built out a successful headline testing program. With the Optimizely A/B Testing Plugin for WordPress you can start testing in just a few easy steps. Try out multiple headlines for each post and measure the difference in performance.

The rise of headline testing

Love it or hate it, headline testing has proven to be a necessary tool for media sites to use in order to stay competitive in the quest for clicks. In a post on Quora when asked what tool Upworthy uses to test their headlines, Adam Mordecai, Editor-at-large writes, ”We have a custom click testing system that I call the magic unicorn box which we built internally. I can’t say how it works. All I can say is that I put my headlines in the box, and the magical scores come out, and then I make decisions based on that.”  Think of Optimizely’s WordPress A/B Testing plugin as your very own magic unicorn box.

We had one goal when developing the Optimizely WordPress plugin which was to give users the ability to create headline variations, view results and launch winners, all without leaving the software authors and editors are already familiar with. Additionally, once you have the plugin installed, you can also start to test any other part of your site using the Optimizely platform. Please visit our knowledge base for full instructions on how to install and use the plugin.

A screenshot of the Optimizely WordPress headline testing plugin.

Don’t have WordPress but still want to do headline testing on your own CMS? Don’t worry, Optimizely has you covered! Download our technical documentation to walk through the process of creating your very own headline testing tool built on the Optimizely REST API. It covers the process we used to create the WordPress plugin, and all of the things you need to think about when AB testing your headlines.

Headline A/B testing best practices

When starting to experiment with headlines on your site, there are a few guidelines we recommend you follow to get the most out of your testing experience:

1. Don’t create clickbait!

Image source: Daily Beast

Wait, I thought that is what headline testing is? A good headline informs the user just enough to want to click the link and read more about the article. It is also safe enough so that other people feel comfortable sharing it. If your headline leads to an article that does not translate into meaningful engagement, the user will quickly lose trust in your site and trust is very hard to get back. Facebook also is cracking down on sites that only produce clickbait. If your stories start to become filtered by the newsfeed, it will be very difficult to recover.

This is why in the plugin, we include the engagement goal along with views to the article as a meaningful metric. It’s important to look at both metrics when making your decision. Even though a headline may have received more clicks, you might want to think twice before launching that winner if your overall engagement declined.

2. Be mindful of the number of headlines to test.

Upworthy famously says they write at least 25 headlines for every story and test them rigorously. If you look at it from a statistical standpoint, you would need a lot of traffic to have statistically confident results that show one headline is beating the others. As you can see using a statistical calculator, it can take up to 10,170 visitors PER HEADLINE to become statistically significant. That is 254,250 per experiment if you are testing 25 headlines. Some sites can have that many visitors in a few minutes, others it might take a few weeks or months.

Also, it’s hard to determine an accurate baseline since every single article will have a different baseline to test against. The nice thing about using the Optimizely plugin for headline testing is that we calculate the results for you so you don’t need to keep track of too many numbers. So while having 25 headlines for every article might make sense for some, try starting small and work your way up to it.

3. Look for big wins.

You can always find a winner the longer you run a test. However you should be looking for big wins meaning lifts of higher than 15-20%. To put this in perspective, let’s change the lift we expect to detect from 20% to 5% and the confidence level from 95% to 80%. You would need 204,914 visitors, PER VARIATION! If you don’t see a big win on a headline, simply move on to the next post.

4. Test first, then share on social media.

The more traffic you get from places like Facebook and Twitter, the more important it is to have the most optimal headline on those services. Facebook and other social sites do not allow for testing headlines on their platform so for posts that are not time sensitive, test on your site before sharing on social.

5. Be experimental.

Some of the best headlines that get the most clicks are sometimes the ones you don’t even think have the possibility of being a winner. This is what makes testing and experimentation fun! As long as you are not creating clickbait (see above), any idea should be tested. You never know what you might find. Here’s a smart post from CoSchedule on writing better headlines.

6. Involve others in the organization.

Generally a good idea for any testing program: be sure to involve others. You don’t have to be the only source of test ideas, and you will quickly find that everyone has an opinion of what will make a good test or headline. Try making it a game or a testing hackathon—whoever creates the most winning headlines for the month gets a prize. It helps generate more ideas, and a little healthy competition never hurts.

Headline testing can be a fun and easy way to create meaningful lift improvement in your key performance metrics while involving the whole organization. Start small and look for big wins without falling into the clickbait trap. You will quickly see how testing every headline is a great way to increase clicks, engagements and social shares and create a culture of testing and experimentation needed to survive in this fast-paced digital world.