If someone is doing something, you can ask them what they're thinking at that point and how they felt after they completed a task or weren't able to.

You can also ask them a little bit about what their expectations are, if they are to go to this other area of the experience that you're testing a prototype for but you haven't fully fleshed out yet. 

It’s a great way to see the physical cues and the body language when you're able to do screen sharing.

And these sessions are really easy to schedule. You can talk to someone in California and then 10 minutes later you can be on the phone with someone in Michigan. 

Also, there's that consistency of the moderator that you don’t get when you have to break up the team and different people will be moderating different sessions. 

You get great results still, but without that consistency, it would seem like you're getting a little bit of hearsay. 

When you start to look at the data, you wonder what exactly happened in that conversation. 

Was the user asked this follow-up question? There may be information that you think could be useful but wasn't captured.

UNMODERATED TESTS

However, there are times when you need to talk to a lot of people and that kind of setup is just not feasible anymore. 

There may also be a specific task that you're trying to get specific measurements on. 

While you can do that through remote moderated testing, it’s going to take some extra footwork to be done.

That's where unmoderated platforms come in. 

Essentially, what you do is create a script for what it is you're trying to test and set all the parameters up. 

Of course, everything has to be tied back to your business goals for your product, whether it’s an app or a website on desktop or tablet or mobile.

Introduce the variants that people are going to be having. 

For example, your script could start with: “Thanks for taking your time out of your schedule. We're going to be asking you a few questions to help us improve the experience of this website or app.”

And then, give people the exact instructions for the series of tasks that you're going to give them. 

It could be: “Go purchase these tires. And when you're just about to enter your credit card data, stop.” 

That could be one task. 

Oftentimes, you’ll also need to include this reminder in your script: “We'll be capturing your audio and video as you're going through this process. So, please remember to share your thought process and your frame of mind as you're going through the steps.”

The big disadvantage of the unmoderated test is you're not there to probe and ask those questions. 

But it's helpful to make sure that you're prompting people of several points throughout the script—tell them to share their thoughts—because people forget. 

Maybe things are working very well and they're excited to just finish that they're forgetting to tell you what they're thinking about all of this. 

You won't know that until you get the final video.

Most of the available platforms for unmoderated tests have user panels that you can recruit from. 

This can be international and you can focus on a particular continent or region if you want.

What you can do is set up demographic controls. 

For example, maybe you want to look for men and women in the Northeast who own a car and have owned it for the past two years. 

You can set all of that up and then the platform will look out at its user panel and only recruit those people who fit those criteria. 

They will be the ones who get the link to go to your test. Once they click the link, they can proceed to take the test.

One of the big advantages of using this method is it makes your analysis much easier.

If you’re recruiting about 60 people for a study, watching the results of all of those videos in order to capture that information is really a lot. 

But by setting up a remote unmoderated test, you can literally have 60 videos in a matter of hours—that would probably take a couple of weeks to do moderated either remote or in person.

There are many other benefits. 

You can schedule people to take the test at the same time. It doesn’t matter if they overlap. 

There’s no need for lunch breaks. 

Travel time is not a factor. 

If somebody doesn’t show up, you won’t have to sit there for an hour with nothing to do. 

All these problems are eliminated. 

And you just recruit as many people as you need. With in-person tests, you usually wind up with only 15 subjects.

Another great thing is that you pay panelists less to participate in these remote tests. 

It makes sense because you have to pay more to motivate somebody to come to you for an in-person session. 

Furthermore, with the unmoderated test, they can do it whenever it’s convenient. They don't even have to plan it.

SOME TOOLS YOU CAN USE

It can be cost effective in that way, although with most of these tools, there is a barrier to entry with membership fees which can be pretty steep. 

There are many different products out there to choose from and they vary in strengths and weaknesses. 

But for those tools where you have the option to do the unmoderated test, UserTesting.com is the one that first comes to mind because they are sort of the gold standard. 

The big downside is that at a certain point, they realized how good they were so they jacked their prices up significantly, which some companies can afford and some just can't.

Validately has an annual membership fee as well, but it's a little bit more palatable and it has a great testing panel. 

UserZoom is also great at setting different types of tasks and units of measure that you can use in your research.

While those are all promising, you can try looking more into utility tools where you actually don't have to pay that annual membership and just go month to month. 

We’re seeing more and more of these pay-as-you-go models coming up and changing the landscape. 

What’s important is that the tool you choose allows you to quickly generate some prototypes and easily get feedback on the general viability of your ideas.