author placeholder

Leonid Pekelis

Finishing a PhD in Statistics from Stanford, Leo is Optimizely’s first in-house statistician. He is passionate about empowering anyone to reap the benefits of experimentation to make informed decisions, whether they're steering organizations or choosing pizza toppings.

Blog Posts by Leonid Pekelis

March 04, 2015

Bayesian vs Frequentist statistics

Just like a suspension and arch bridges both successfully get cars across a gap, both Bayesian and Frequentist statistical methods provide to an answer to the question: which variation performed best in an A/B test? Historically, industry solutions to A/B testing have tended to be Frequentist. However, Bayesian methods offer an intriguing method of calculating experiment results in a completely different manner than Frequentist. In the world of statistics, there are devotees of both methods—a bit like choosing a political party. In this post, we’ll cover the benefits and shortcomings of each method, and why Optimizely has chosen to incorporate elements of both into our Stats Engine.
January 20, 2015

Statistics for the internet age: The story behind Optimizely’s new stats engine

Classical statistical techniques, like the t-test, are the bedrock of the optimization industry, helping companies make data-driven decisions. As online experimentation has exploded, it’s now clear that these traditional statistical methods are not the right fit for digital data: Applying classical statistics to A/B testing can lead to error rates that are much higher than most experimenters expect. We’ve concluded that it’s time statistics, not customers, change. Working with a team of Stanford statisticians, we developed Stats Engine, a new statistical framework for A/B testing. We’re excited to announce that starting January 21st, 2015, it powers results for all Optimizely customers. This blog post is a long one, because we want to be fully transparent about why we’re making these changes, what the changes actually are, and what this means for A/B testing at large.