Challenge: Losing Variations

Lin leads an optimization team for TravelAsia, a website and app that helps visitors create and coordinate travel across multiple Asian countries in one trip. Both the website and app currently feature a search function at the top of the homepage. However, Lin’s team found that most of the information people searched for was accessible in some way from the main page of the website. This information was clearly visible as long as people actually looked, and by accessing this information directly from the main page, it’d take fewer clicks to get to what visitors needed. Because of this, they considered the search bar a distraction that didn’t encourage people to look at the whole page to get what they needed. In order to encourage visitors to review more of the page to come across what they needed organically, Lin’s team created an experiment that moved search bar from the top of the page to the bottom to encourage people to scroll entirely down the page to find the information they need before trying to search for it.

After a few weeks, the experiment reached statistical significance and Lin checked the results. The experiment was a huge loss. In the experiment group, they had a nearly 50% increase in dropoffs compared to the control group.

Before reading how Lin and his team handled the loss, take a minute to think about how you would handle the situation.

What should Lin and his group do next?

  • Alter the experiment and try it again to get more information about why the first experiment failed.
  • Visitors obviously didn’t like the new setup, so return the search bar where it was and experiment on something else.
  • Analyze the results before deciding what to change for the next experiment.
  • Immediately re-run the experiment to confirm the results. Maybe the timing was poor for this experiment for some reason?

What analysis tool would most effectively help Lin’s team understand what caused the failure?

  • Funnel Report
  • Heat Maps
  • Site Search
  • Scroll Depth

What else could Lin’s team do to learn more about their results?

  • Review timeframes for each dropoff
  • Segment their results
  • Check to ensure there were no technical issues that caused the results to be skewed