How Many Users Is Enough?


In discussing user testing, there's a question that comes up a lot. What can one realistically glean from a few (say, 5) individual usability tests? After all, a small sample set is not statistically significant, and there's no assurance that those individuals are representative. Some people indicate that they would only be comfortable with a large sample set, which can be quite time-consuming with an activity like usability testing.

While Nielsen et al have shown empirically that in many ways the optimal number of users per test is 5 or so, we like to provide a more quotidian response as well.

Map

Imagine you visit a foreign country, say China or Mongolia or Egypt, and find yourself in conversation with one or two local individuals. Can you learn anything from these conversations? After all, who’s to say they are representative of prevailing attitudes?

Of course, the answer is yes, you can learn something, because the exposure to a different perspective is often educational, and may contain many “Aha” moments. This is particularly true politically, because we approach global political issues with a lens shaped by our media, political environment, and so forth, which is quite different from that of other countries. That is, our mental model is quite different from that of others. Similarly, albeit usually not to the same degree, we design websites with our mental models, which are not necessarily the same as that of our target audience. After all, we are experts both in the domain and in the technology.

Even a few narrated videos can be instructive in showing us how well our mental models and information architecture map to what is expected by our audience. It may not seem that scientifically “valid” to make judgments based upon a handful of data points, but what we’re doing is not “scientific,” it’s softer and more creative and deductive.

 

Related reading: