The Stats Engine is Optimizely’s proprietary system that combines innovative statistical methods to give you trustworthy data faster. In contrast to traditional statistics, which assumes that you will only evaluate your experiment data at one point in time, at a set sample size, sequential testing is designed to evaluate experiment data as it is collected. Sequential testing makes decisions as soon as you see a winner and they can be stopped any time with valid results.
Experimenters rarely have a fixed sample size available, and their objective is typically to get a reliable inference as quickly as possible. Stats Engine meets these goals with an implementation of sequential testing that calculates an average likelihood ratio—the relative likelihood that the variation is different from the baseline—every time a new visitor triggers an event.
False Discovery Rate Control allows for many goals and variations with guaranteed accuracy. Reporting a false discovery rate of 10% means that “at most 10% of winners and losers have no difference between variation and baseline,” which is exactly the chance of making an incorrect business decision. With Stats Engine, Optimizely reports winners and losers with low false discovery rate instead of low false positive rate. As you add goals and variations to your experiment, Optimizely will correct more for false discoveries and become more conservative in calling a winner or loser. While fewer winners and losers are reported overall, an experimenter can implement them with full knowledge of the risk involved.
When combined with sequential testing, false discovery rate control provides an accurate view of your chance of error anytime you look at results. The control gives you a transparent assessment of the risk you have of making an incorrect decision. This means you can experiment as many goals and variations as you want with guaranteed accuracy.
The Stats Engine updates in real time and will ensure a 95% significance level results every time, so you can be confident you’re making the right decisions for your company.
You can learn all about our Stats engine by visiting optimizely.com/statistics.