This is part two of an optimization case study with Secret Escapes, a flash-sale luxury travel company. In part one we covered how Secret Escapes used Optimizely to experiment on their iOS app onboarding flow, double signup rate, and generate positive ROI from paid mobile marketing.

In this post, we’ll dig into another series of experiments that are driving continual increases in new user signups and revenue. In both experiments, Secret Escapes is testing the impact of personalized experiences.

Experiment 1: Personalization for PPC Landing Pages

The company had high click-through rates on their search marketing campaign ads and directed all of this traffic to a single Pay Per Click landing page. They saw an opportunity to convert more paid traffic on this page.

Tom Evans, a product manager and A/B testing lead at Secret Escapes, hypothesized that mirroring copy on the landing page with the copy on the ad would improve conversion rates. In other words, they tested the impact symmetric messaging between ads and landing pages had on the rate of new user signups.

The test stems from the idea of matching user intent with expectation. Entering a search in Google is a clear display of user intent. The hypothesis is that a symmetric experience from the ad to landing page —one that clearly matches that search intent — will make people more likely to take the desired action on the page.

Secret Escapes “spa” PPC ad in Google.

Tom ran this test for their “spa” AdWords group first. He tested two different personalized pages against the control landing page with no personalized elements:

In variation 1, he changed the image to a beautiful spa pool and changed the sub-head to include the keyword, “spa”.

In variation two, he kept the beautiful spa image and made the headline copy symmetrical with the ad.

The test ran on a sub-set of traffic: users who came in using spa keywords. In Optimizely, he set this up by targeting the test to a query string used by a number of the spa ad-groups. View variations side-by-side.

Experiment results

Secret Escapes saw a 32% increase in conversions in new user signups from visitors who received the symmetrical messaging in variation 2, the spa image and symmetrical headline.

Rather than a one-off, this result translates into a proportional increase in new users for Secret Escapes at no added cost of AdWords spend. Next, the team will test symmetric messaging on different ad groups.

Experiment 2: Personalization for Logged-In Users

Using search data to personalize vacation listings

For a logged-in user, visiting the homepage (secretescapes.com) used to be a long list of vacation listings to everywhere from a casino in Las Vegas to a beach in the Dominican Republic. Other than timeliness of the deal, there was no rhyme or reason behind why a particular listing appeared.

Original logged-in homepage experience for Secret Escapes.

“Whilst we curate and hand-pick every offer that features on the site, we don’t do a great job at creating a personal list for our users,” Tom wrote in a post on the Secret Escapes’ tech blog.

Not anymore. After running an experiment that increased conversions, it’s more likely that logged-in users will see homepage listings that are tailored to the types of vacations they have searched for previously.

“Personalisation is one of the things I really expect to blow up in 2015,” he continued. “With so much data lying around about our customers, and their behaviour, we now have a great opportunity to optimise their experience by providing more relevant sales and information.”

Using Optimizely and a hack-day, Tom implemented an experiment that would prove the concept and test the benefit of personalization of homepage listings.

The test re-arranged the sale listings so users who arrived to the site from a PPC keyword of “New York” or “All Inclusive” saw relevant sales promoted to the top of the listings, as seen below:

Personalized search on Secret Escapes homepage.

How did he do this?

Tom knew he could implement the experiment with Optimizely and jQuery Cookies. He explored the idea of basing the experiment around cookies that would detect three different bits of information about a user:

1. Last 3 search terms that brought a visitor to secretescapes.com.
2. Most visited type of sale for that visitor. If a user shows a certain propensity for specific types of sales promote those sales higher in the list.
3. PPC keyword that acquired the visitor. If visitors click an ad for “spa weekend” and sign up for Secret Escapes, then promote sales around those keywords for the next X days.

The test became complex when Tom considered which experience to serve users who match all three criteria. So for V1 of the experiment he ran with a single cookie type, PPC keyword acquisition. This was the simplest of the implementations, as it wasn’t a dynamic or changing cookie. Once they had signed up, the cookie would remain the same until it expired.

Tom set the Optimizely experiment to run for 100% of users on the appropriate URLs.

The impact of personalized listings

The initial results have been incredibly promising, Tom wrote. The personalized experience has resulted in positive lift for all of their KPIs. For the New York keyword they saw the following increases after running the test for 140 days:

• Clicks to sale from listing page: +24.4%
• Clicks to purchase page: +30.4%
• Completed purchases: +210%

This experiment is the tip of the iceberg for further personalization on Secret Escapes. It opens up many exciting questions and other possibilities to tailor the experience to specific segments of users.

Interested in learning more about optimizing paid search campaigns? Download our eBook on driving more revenue from PPC with A/B testing.