Sometimes, when you check your experiment at the end of the projected time-to-results, your experiment hasn’t reached statistical significance. While you know that inconclusive experiments can provide valuable information, you also know that waiting longer will help you gather more data. So how long should you wait?
To answer that question, ask yourself what the potential impact of this experiment is, compared to the next experiment you might run. In certain situations, you may wish to take action on an experiment that’s run for the projected time but hasn’t reached significance. This is a business decision you’ll make based on the results page data and your business goals.
There are three key factors to consider in these situations.
First, check your visitors remaining number. This is an estimate of how long your experiment has to run to reach significance, based traffic, conversion rate, and number of variations.
Wait for statistical significance if you haven’t reached the number of visitors predicted in your experiment plan or if your number of visitors remaining suggests you don’t have long to wait. However, you may want to declare the experiment inconclusive if you’ve exceeded the number of visitors you planned for, or if your visitors remaining number suggests you won’t reach significance anytime soon.
For example, imagine that you want to be confident that the 5% lift you see is valid. But your visitors remaining number tells you it’s likely you’d have to wait another two weeks to gather enough data to reach 90% statistical significance. You could keep running the experiment, but it’s possible you won’t see anything change in those two weeks. Ask yourself: is this a “typical” timeframe that can at least give me a solid basis upon which to draw a conclusion? Are there other factors I might need to consider before making a decision to end the experiment, such as a promotion or holiday that might have impacted these numbers?
Together, the data for visitors remaining, your experiment plan, and the lift you expect to see already suggest that the change you made did not affect your visitors’ behavior in a significant way. At this point, it may be best to segment and check your secondary and monitoring metrics to look for insights for the next round of experiments.
You can also check the confidence interval. The confidence interval can help you decide whether to keep running an experiment or move on to another idea. The interval always straddles zero for experiments that haven’t reached significance yet.
If your results are inconclusive, your confidence interval can provide insight when paired with visitors remaining. Consider an inconclusive test with a difference interval from +0.2% to +0.4%. Stats Engine calculates 6,500 visitors remaining before you reach significance. Is it worth it to keep going?
Finally, segment your results! You may find a group that responds to your experiment differently from the average population. Or you might discover statistically significant results for a specific segment of your visitors.
When you learn how to interpret them, inconclusive results can often offer valuable, and sometimes surprising, insight into your visitors’ preferences. Understanding these key factors can really help you to make a confident decision no matter the outcome.