A/B Testing All-Star: John from IXL Learning

Name: John Barth
Company: IXL Learning 
Role: Senior Marketing Analyst
What’s the average age of students who use IXL Learning? Our most popular grade is 2nd grade, so, most of our students are around 8 years old.

John says, “Every problem I had with Google Website Optimizer (now Google Optimize) had been thought through and solved by Optimizely. The difference between the two is like night and day.”

Optimizely: When did you start testing and why do you test?

John: We started testing about a year ago. I had ideas for several different things I wanted to test for awhile, but without an A/B test framework, you can’t properly see the differences between the original page and the test variation, other than taking the plunge and implementing the changes.

We started out with Google Website Optimizer (GWO), but I was disappointed with its limited functionality and difficulty in test deployment, especially that it required the engineers to code the different variations and then put them under different URLs. We were just starting to look at different A/B testing solutions when Julio reached out to us, and the rest is history! Every problem I had with GWO had been thought through and solved by Optimizely. The difference between the two is like night and day.

What is the testing culture and process like at IXL Learning?

Everyone on the marketing and development teams are encouraged to come up with ideas for tests. When we all like the idea, we have our UI designer to come up with a few different variations. Then I create the variations in the Optimizely editor (or get help from an engineer when there are larger UI changes), get approval from the product management team and the VP, and deploy the test. On the long side, it can take a couple of weeks to get everything set up and approved for our more complicated tests. On the short side, I’ve taken an idea from a concept to an actual running test in under an hour.

Where do your testing ideas come from?

I follow some blogs and read about tests other people have run for ideas. I also try to think like a user and try to figure out what I can improve on our pages to improve engagement or continue through our subscription flows.

Tell us about a test that has yielded actionable results.

The membership area on our site directs users between our school subscriptions and our family subscriptions. It’s also the first page in all of our subscription flows, so it’s critical that this page be easy to understand and navigate. The original page had all the calls to action on the right side of the page, including some below the fold, and one column of text across the page. Moving the calls to action above the text, and breaking the text into two columns saw gains across the board for all of our conversion types!

Before

After

What’s your favorite Optimizely feature to use in your tests and what new feature would you like to see?

There are so many it’s hard to pick. My absolute favorite is multiple goal tracking. We have several different types of conversions on our site, so when I run tests, it’s critical for me to see the impact to all of them. Sometimes the best variation will see some decline in one of the conversions at a larger gain in another.

One thing I would like to see that is more of a new product than a feature would be adding A/B testing for e-mails.

Share some best practices you’ve developed with testing.

• Don’t make assumptions about what users will like or not like. Sometimes they’ll surprise you.
• Some of the biggest gains can come from the smallest tests. A lot of the higher impact tests I’ve run have been simple changes in button colors and text.