How many users is enough?

I was recently discussing usability testing and our product with a friend, who asked what one can realistically glean from a few individual usability tests.  After all, a small sample set is not significant, and there’s no assurance that those individuals are representative.  He indicated that he would only be comfortable with a large sample set, which would be difficult to assess using a tool such as TryMyUi.  While Nielson et al have indicated that in many ways the optimal number of users is 5 or so, I tried to provide a more quotidian response.

Imagine you visit a foreign country, say China or Mongolia or the Middle East, and find yourself in conversation with one or two local individuals.  Can you learn anything from these conversations?  After all, who’s to say they are representative of prevailing attitudes.  Of course, the answer is yes, you can learn something, because the exposure to a different perspective is often educational, and may contain many Aha moments.  This is particularly true politically, because we approach global political issues with a lens shaped by our media, political environment and so forth, which is quite different from that of other countries.  That is, our mental model is quite different from that of others.  Similarly, albeit usually not to the same degree, we design websites with our mental models, which are not necessarily the same as that of our target audience.  After all, we are experts both in the domain and in the technology.

Even a few narrated videos can be instructive in showing us how well our mental models/information architecture maps to what is expected by our audience.  It may not be scientifically “valid” to make judgments based upon a couple of data points, but what we’re doing is not “scientific”, it’s softer and more creative and deductive.