Detecting Incorrectly Implemented Experiments

Has the outcome of an experiment ever gone so strongly against your intuition that you doubt its correct implementation? This is certainly a possibility, as correctly implementing an online experiment is fraught with data quality challenges. There is a considerable amount of engineering and processing of data before it reaches the analysis, leaving plenty of room for bugs and biases to creep in. One of the strongest signals of an incorrect implementation is a sample ratio mismatch (SRM) - when the number of users assigned to each variation differs significantly from what is expected under the intended random allocation.

This talk will:

  • Demo a novel SRM test which allows experimentation teams to rapidly detect if there is a bug in the implementation, even while the experiment is running, allowing developers to quickly fix the underlying issue.
  • Provide an introduction to both the mathematics of the SRM test as well as the newly open sourced library which developers can immediately integrate into their platform.


Michael Lindon

Staff StatisticianOptimizely

Watch Now

Welcome back. Not you?


Sorry, there was an error. Please try again.