User Testing: A Pillar of Great Experiments

If this temple in Agrigento Sicily island in Italy was an experiment, then user testing would be one of its pillars .

If you’re working on optimizing your conversion rate, chances are, you’ve already done some testing on your website. Maybe it’s A/B testing or multivariate testing, or maybe you’ve run a heat map or two. These are all tools available to you to deliver the best experience possible to your visitors. In this article, we’re going to talk about another one, user testing: what it is, and how you can use it to come up with really good hypotheses for your experiments.

The definition of user testing and how it benefits experimenters

In a nutshell, user testing means watching over someone’s shoulder as they use your website or app. They’ll speak their thoughts aloud while they interact with the screen, and you literally watch. You can see where they get stuck, confused, frustrated or delighted. It’s an excellent way to get qualitative feedback on the user experience of your site.

Marketers need this feedback because we see our own websites, apps and products so often that it’s easy to become blind to their problems. We need to get fresh eyes on our site so we can create hypotheses for what we need to change and improve.

Here’s 1-minute video example of what you’ll see when you run a user test.

The logistics of user testing

You can run a user test in-person or remotely. Some companies have usability labs in their office. They’ll invite folks to take a test in the lab, and observe them using the site. In-person testing has a few advantages: you’re right there to see the person’s facial expression and body language.

On the other hand, if you choose to do a remote test, you’ll watch a video of the user’s screen and listen to their voice. Advantages: you don’t have to worry about setting up a lab, or recruiting the test participants yourself. Both methods have their pros and cons, so you’ll have to decide which makes sense for you.

There are five steps to running user tests and turning the results into great experiments. 

Step 1: Decide what you want to learn

Rather than trying to find every single problem with your site in one round of testing, hone in on a specific set of questions, or a flow you want to improve. You’ll get a deeper understanding of that particular scenario, and then you can test other processes in the next round.

This is an important step. Make sure you spend enough time thinking through exactly what you’re hoping to learn from your user test.

One thing you might want to learn is whether your site has any major usability problems.

• For e-commerce: watch users attempt to find an item, add it to their cart, and go through the checkout process. Take notice of any issues they encounter along the way.
• For SaaS: watch users go through the onboarding flow.
• For media: ask people to find an interesting related article and watch their process.

You can also employ user testing to learn why A outperformed B in an A/B test. If you are truly stumped by the results, run a user test on the A and B experiences.

User testing is a great method to learn what users think about your competitors. It’s especially easy to do competitor testing remotely. Users never have to know which site is yours and they won’t feel obligated to spare your feelings.

Step 2: Pick your target audience and recruit

Once you’ve decided what you want to learn and the area of your site to test, decide who to test with. Think about your target audience and any personas you’ve built: whose feedback is most pertinent to you?

If you’re testing in person, you will need to recruit those people (offering an incentive to take your test is important!). If you’re using a remote testing tool, you’ll just need to identify your target demographics.

Here’s an example. Let’s say you work for a high-end jewelry company where the target market is women ages 35-50. You should recruit people who fit this exact profile. You could also get more specific and refine the user base to women ages 35-50 who frequently shop online. Do this with a screener question like, “In the past year, how many times have you shopped online?” and accept people who choose “Four times or more.”

How many users do you need to test? Good question. Since user testing is a method to gather qualitative feedback and not statistical significance, you don’t need many people to participate. Nielsen Norman Group says that 5 users is all it takes for big insights.

Step 3: Write unbiased tasks and questions

Next, you’ll want to choose a series of tasks for users to complete. Depending on what you’re trying to learn, they can be open-ended, like, “What is the first thing you would want to do on this website? Try to do that thing now,” or specific, like, “Try to find a hotel room around $200 in New York City next week for two guests.”

It’s a good idea to follow tasks with questions about whether the user thinks he completed the task successfully, how easy or difficult it was, and how he felt about the process.

While it’s always nice to hear positive feedback, it’s more helpful to learn what you can improve. You’ll need to be careful that you aren’t skewing the results in your favor (in other words, fishing for compliments). For example, here’s a multiple choice question we recently saw on UserTesting:

What word would you use to describe the shopping cart page?

• Clean
• Clear
• Well-designed
• Informative

I’m sure the results from that test will make some designers very happy, but it doesn’t teach them a whole lot. A better question would include some negative options as well as “None of these.”

Step 4: Watch, learn, and take notes

Time for eagle eyes (and ears).

Once you’ve setup your test, it’s time to watch users interact with your site.

Observe exactly what users are doing when they have emotional reactions like expressing frustration, confusion or delight.

Pay attention to any tasks that took a long time to complete. In ideal cases, users should be able to quickly and easily find what they need. Tasks that take a long time should reveal opportunities for improvement.

If you include multiple choice or rating scale questions in your test, note of any trends that arise and investigate the outliers. If four participants think a task is easy, but one thinks it’s extremely difficult, focus in on why it was difficult for that one person.

Step 5: Look over your results and A/B test

Soon (hopefully immediately) after your testing sessions, go through all of your feedback and sort it into recurring themes. Create a list of key takeaways from each user, then turn that list into a set of concrete ideas to either test or change. If the change is very slight it might not be worth A/B testing due to the time it will take to reach statistical significance.

Add user testing to your set of optimization tools

You can use the insights you’ll gain to form the basis of your conversion rate optimization plan. I guarantee you’ll get some great ideas for your next A/B test. You’ll probably find a few things you can easily tweak on your site to boost conversions. And best of all, you’ll learn more about what’s going on inside the heads of your users.