Quiz: How Much do You Know?

Before we start, let’s see what you already know with a short quiz. Select the best response to each question. When you've answered all the questions, select the submit button to check your answers.

Raj and his team have been running small experiments with a good success rate for the last few months. They just completed their first significant A/B experiment that included a number of major redesigns to their product pages and, even after reaching statistical significance, their results are inconclusive. What should they do first?

  • Check the parameters and re-run the experiment. Maybe the experiment was set up wrong or ran at a bad time.
  • Segment their results to see if there were wins in any area. It’s possible the changes were effective in different segments that could give insight into the results.

What are some reasons why an experiment could come back as inconclusive?
1- Sample size is too small
2- Don’t have enough traffic
3- Experimented with something visitors don’t care about
4- Change is too small to make a difference

  • 1, 2, 3, 4
  • 2, 3, 4
  • 2, 4
  • 1 only

What is one way you can prevent an inconclusive result?

  • Make bolder changes
  • Run experiments longer
  • Research market standards
  • Avoid redesigns without functionality changes