If you want to understand how a lion hunts, don’t go to the zoo. Go to the jungle.
There are a lot of ways to do usability testing, and all have their own merits. But in terms of getting the most reliable, accurate feedback for your website, unmoderated remote user testing has distinct advantages over the alternatives.
Unmoderated remote user testing vs on-site user testing
On-site user testing, in addition to being many orders or magnitude more costly, also produces less reliable results than remote testing. On-site testing is quite artificial – you are putting users into an unfamiliar environment, making them use your unfamiliar devices rather than their own, and physically watching over their shoulders, directly interfering with their experience.
There is no way for the user to slip into their “normal mode” – browsing around the site naturally as though they were doing it on their own volition, to their own ends. They are in “testing mode,” and constantly aware of it – something that is not helped by the presence of the moderator.
With remote user testing, the users perform their test in a typical, representative environment using their own device on their own time. While some elements of the test will still unavoidably be artificial, it will come much closer to replicating a natural user experience, and that means the feedback you get will be much more relevant to how real people are using your site “in the wild.” The most expressive, clear, and eloquent feedback is still useless to you if it doesn’t reflect the reality of your product’s user experience.
In addition, with remote user testing you have access to an enormous and diverse pool of users all over the world. You are not limited to users in your local geographic area, unlike with on-site testing. With such a wide pool to draw from, you can get much closer to finding people in your specific target audience, whose feedback is far more valuable to you.
Unmoderated remote user testing vs moderated remote user testing
Moderated remote user testing gets rid of some of the issues with on-site user testing, but the very presence of a moderator, while it may seem appealing, actually decreases the value and accuracy of the results of your study.
People have a natural tendency to try and please others and make themselves look good, and this trait always comes into play (even if often subconsciously) when users are interacting with a moderator. In trying to please the moderator and do and say the right thing, the user stops representing their true experience with the product and is instead just saying what they think the moderator wants to hear.
Removing the moderator from the picture makes it easier for the user to be brutally honest because that human connection is no longer interfering. There is no name or face for the user to empathize with, or feel an attachment to; when they voice their thoughts and reactions, it is as though they’re talking to themselves, not talking to another individual.
Moreover, moderating an on-site user test is very hard to do especially if you are close to the product being tested. The key to professionally moderating a test, in part, is to keep your mouth shut most of the time. You must be able to let the user stumble around and find his way without jumping in to save him. This is often very difficult to do for owners, developers, or designers of a website.
More reliable usability insights
It’s tempting to want more control over your testing process, and moderation seems to offer that additional control and oversight. But in the end, it is more harm than it is worth; even without the artificial environment created by on-site testing, moderation of remote user testing distorts user feedback and makes the results less reliable.
The key to great usability testing is not control. It is authenticity. Only unmoderated remote user testing comes close to realizing a genuine user experience that can provide relevant and reliable insights.