Creating Excellent Hypotheses with Website Analytics and User Research

This is the final installment of our three-part case study for Move, Inc. — the operator of Realtor.com — and how it 1) organizes its optimization team and 2) focuses on a quantifiable goal to drive internal buy-in for testing.

For an optimization team to be successful in reaching an aggressive revenue goal, both quantity and quality of experiments must be high.

To support the Move team’s focus on testing velocity, a strong backlog of experiment ideas is essential. In the first and second installments of the Move, Inc. team’s story, we learned that Suma Warrier and her optimization team are located in the same department as the company’s analytics and user research teams. This has helped the team build an ideation process grounded in both data and user feedback.

Turning qualitative and quantitative data into action

Suma and her team ensure that all test ideas are backed with a hypothesis, supported by either quantitative or qualitative data. Analytics and research provide them with a fundamental understanding of how users behave; running experiments helps them to understand if they’ve alleviated points of friction for their users, getting them the information they need faster, or keeping them engaged longer.

Analytics and user research complement each other in the ideation process. Suma’s team also circulates an internal request for experiment ideas to ensure that other teams have input into the optimization process

Suma and her team used a simple form to collect experiment ideas from teams internally to add to their backlog.

Analytics also helps Suma’s team to quantify the impact of a test. Are users dropping off close to a conversion point? How many users will be impacted by the test? What is the potential lift if the variation is successful? The greater the projected impact of the test, the higher it will be prioritized in the experiment backlog.

Here are a couple of thoroughly vetted experiment ideas that Suma and her team have tested.

An analytics-informed experiment: right rail content

In this experiment, Suma and her team built an experiment hypothesis with intelligence collected from their web analytics.

On the search results page, conversions on right rail content were relatively low. The Move team wanted to find a way to engage visitors who might not be ready to click on listings, but are looking for other information that will help them progress towards choosing a home. They hypothesized that surfacing links to intermediate steps in the home-buying process would improve search engagement with the right rail and make conversions in later stages of the conversion funnel more seamless.

The change: Suma and her team added a module (outlined in red) that highlighted getting pre-qualified and checking a credit score as logical next steps for visitors.

The result: The change in right rail content didn’t improve conversions, but it did provide an opportunity for further testing. The team was able to move on from the type of content surfaced to exploring other elements of the right rail functionality, and how it could improve the visitor experience.

A user research-informed experiment: navigation styling

In usability studies, Move researchers found that when viewing listing pages for homes, it was generally simple to complete an action on the page (saving, requesting details, etc.). When it came time to navigate away from the page, visitors were getting stuck. They wanted to see other houses and view other results from their searches.

The optimization team hypothesized that adding a “Back to search” link would help users understand where to navigate once they were ready to leave a listing page.

The change: A “Back to search” link was added to the navigation elements, next to a breadcrumb. This experiment focused on helping visitors to engage with multiple listings during their sessions, increasing the likelihood that they would advance further down the conversion funnel.

The result: The change positively affected the engagement of visitors on listing and search pages, demonstrating that the researched change made a quantifiable impact on the visitor experience.

By maintaining close alignment and open communication with their analytics and user research peers, the Move, Inc. team is able to consistently produce high-quality experiments that help each group to accomplish and measure their goals while improving the usability of their site and overall customer experience.

To learn more about how the Move, Inc. team optimizes, take a look at their team structure and communication best practices, and how they use a primary revenue goal to focus their efforts.