Forbes Japan: Discover What Comes Next With ‘A/B Testing’

Digital experience optimization is the most significant technology to accelerate a company’s success in an increasingly competitive digital world. Not only do we believe that but so do our customers and the industry analysts within this space. Recently we hosted a webinar with Forrester’s James McCormick, he discussed how A/B Testing and Behavioral Targeting are technologies that provide huge opportunities for businesses and can be used everywhere.

Read Forbes Japan Report entitled Creative Sales. Yosuke Iseki interviews Optimizely as part of the “New ‘How to sell’ feature to uncover alternative products and services that increase customer engagement. According to Forbes Japan — and we agree! — customers are increasingly important in business.

Discover What Comes Next With ‘A/B Testing’

Companies like Amazon and Google consider experimentation the main factor for success, not simply one reason. A certain American firm believes in the power of experiments and wants to make its ’weapon’ widely available for exactly that purpose. 

text by Yosuke Iseki / photographs by Ramin Rahimian

Optimizely

An American Digital Experience Optimization Company founded 2010 in San Francisco and known for developing their A/B Testing product which promotes higher viewing and sales rates compared to other digital products. The photo shows company CEO Jay Larson (right) and former programmer turned CMO Carl Tsukahara (left).

Take a look at the two images in the right sidebar. Both were used on his 2008 presidential campaign homepage when he won the election for the first time. In one image Obama has a self-confident expression and looks straight ahead, brimming with confidence, while in the other Obama is smiling and surrounded by his wife and children.

Which image do you think led to more email signups, and ultimately, more donations?

That’s the question we were posed when we visited the San Francisco offices of the Experimentation Platform, Optimizely. Both the cameraman and myself guessed that the first image with Obama by himself was the winner.

“Unfortunately, you’re both wrong!” said Optimizely CEO, Jay Larson while smiling along with CMO Carl Tsukahara.

“This is the feature that makes our product stand out. Although most people assumed the confident photo of President Obama was the one that drew in more campaign donations, the fact is the family photo beat it by a mile. However, it’s precisely because it’s hard to know at the beginning which one works best that we need to make a variety of people the subject of ‘experimentation.’”

American Digital Experience Optimization Company ‘Optimizely’ was founded 10 years ago by former Google employees Dan Siroker and Pete Koomen. Siroker left Google after 7 years and gained experience supervising web route analysis and online donations for the Obama Senate election campaign (at the time).

Siroker’s use of A/B Testing then brought $60 million in online donations to the Obama administration which had little in the way of funding and tools. These earnings made up 60% of the $100 million in capital they supposedly needed in election funding.

“Which attracts more users, Version A or Version B?” “How much could sales be increased?” Following the Obama campaign, Siroker, who now had a true sense how effective this ‘experimentation’ could be, he set out to bring the power of A/B testing and experimentation to the business world.

Business Decisions Based on Data and Evidence

Optimizely is the leader in the Digital Experience Optimization space with 26 of the Fortune 100 as customers, including Microsoft and IBM and other big companies such as VISA and the New York Times. They have 2 products, one which can be used to run experiments on the web, and a Full Stack product which can be used to run tests in apps and on the backend.

With the former, if customers set up two to three variations (choices), the Optimizely platform will deduct the ‘winner’, which makes utilizing analysts and data scientists redundant.

For example, editors of the American Wall Street Journal use this platform for ‘Headline Testing’, which increases the viewing rate by changing the title of articles. Also, customers are able to choose the subject of their experimentation. If a retail firm were to launch online sales, they could narrow their test range to 5% of all traffic. Moreover, the ‘winner’ can be identified in a short space of time, so it is also possible to adapt that variation to the whole. Banks, insurance companies and so on, can pitch financial products pinpointed at wealthier customers. The latter full stack product is a clear choice for in-house web or app solutions and can also be applied to loT products.

As digital products will increase dramatically in the future, the market will continue to expand. However, Larson explains that his company’s product “applies scientific techniques to decision-making in business” which is more significant than anything else. In the past, even if the idea was there it wasn’t easy to test it. There were not enough ways to access the data either. However, the world has completely changed due to digitalization and these days everyone has vast volumes of data. As a result, ‘experimentation’ at a precise level has become possible.

Tsukahara added, “We explain to potential customers that our aim is ‘removing the ‘guesswork’ from the digital which means sales can be improved by using data obtained from experimentation rather than guesswork when creating websites or apps. Also for larger projects, risks can be reduced by using experimentation.”

In the digital business world as well, sales staff make decisions based on instinct and experience, but “the data for that premise is insufficient”, Larson points out.

A/B Testing can address those problems but it’s hard to say the potential is sufficiently understood. Still, Larson does not try to mask his confidence. “I firmly believe this is the next most essential aspect in business management. I’m sure companies that succeed will be smoothly implementing ‘experimentation’ from now on.”

Why is A/B Testing effective?

A/B Testing presents several versions of a web page or app to users and determines which one shows the better performance. By comparing the current version (control) to another pattern (variation), this method can readily gather more detailed data in response to the user’s needs.

Optimizely CMO Carl Tsukahara says, “In Japan where the population is expected to decline, it’s bound to be difficult to employ data scientists in large numbers. From a technological perspective, our product is not dependent on staffing resources and is also capable of deriving the ‘answer’ without guesswork.”