USABILITY TESTING: ARE 5 PARTICIPANTS ENOUGH?

“You Only Need to Test with 5 Users” – the statement which so many people swear by! Why and how did 5 become the magic number for usability testing?  The Common Industry Format (CIF, part of the U.S. Department of Commerce) for Usability Test Reports advises a minimum of eight participants! And, there are a few more widely accepted variations.

It is true that it takes 5 or 8 users (provided they are from your target audience) to test your application in order to identify a good majority of usability problems. However with the more commonly (mis)quoted 5-user rule, there are some caveats and it’s important for people who research, to clearly understand those.

Let’s revert back to the caveats in a bit. The question that the above discussion begs clarity on is “How many people to test with?”.

Short answer: It depends.

“Depends”? … on what?

That is one of the struggles UX Researchers have because there is no fixed and correct answer.

A number of factors come into play when deciding sample size. These could be one or more of the following questions that may help to decide the sample size for usability testing:

  • What exactly is being evaluated? Design concepts, interactive prototypes, working products, etc.
  • At what stage of the project is the usability evaluation being done?
  • How many times the usability testing on the same product/material has already been performed? 
  • Is iterative usability testing on the same product/material already planned? 
  • What is at stake? How is the outcome supposed to be used? Initial Design refinement vs Product launch decision, etc.
  • Who are the stakeholders involved? In the case of business decision-makers, the sample size must match the organization-wide (undocumented) magic number, below which the outcome might not be considered credible.
  • What type of usability testing is being performed? Qualitative (focus on usage insights) or Quantitative (focus on usability metrics)?
  • How complex is the study? Comparative / Benchmark studies need some quantitative results with statistical significance for decision-making.
  • What is the budget and timeline?

A factor that has been out of consideration in the above discussion is about how the usability study is being conducted. A selection between remote or in-person moderated usability testing may not affect the sample size significantly.

However, selecting an unmoderated usability study calls for a slightly larger sample size. To understand this better, the unmoderated remote usability studies are performed without the supervision of a moderator. These studies make use of a usability testing software or user research platform on which the item to be evaluated is uploaded, users taking these studies at their convenient time. While the usability feedback gathered from this method has less bias due to a natural context of use, a few factors to be taken into account. Researchers may end up rejecting a small percentage of responses due to technological limitations, distracted participants or hard-to-comprehend voice feedback. Based on our experience with thousands of usability tests conducted on the UXArmy platform, we propose that the sample size for unmoderated remote usability testing must be 5 – 8% larger than the same study when conducted under supervision.

Back to demystifying the 5 user folklore, Jakob Nielsen is often quoted with his advice that “Five participants will discover over 85% of the problems”. Taking one step at a time, the NN group website clarifies that the 5-users statement was applicable to a specific context. 

The Nielsen Norman Group website recommends applying statistical significance to calculate sample sizes while doing quantitative usability studies. For qualitative usability studies, there are a number of assumptions and a recommendation for doing iterative usability testing with 5 users. It is assumed that the nature of the usability study is qualitative and a number of follow-up usability testing iterations shall be made to thrash out the remaining usability problems in a Design.

 

The bottom line is, it takes more than 5 users in case your product usability is already good. It’s true that a larger sample size would be needed to find those hard-to-discover usability problems. So if usability testing is a practice at an organization and has been happening for the interface/product in question throughout the development, a larger sample size would be needed towards launch to ensure the remaining usability problems are found and if needed, fixed before the product release.

More participants don’t always mean that the research will be of a higher quality. Recruiting the correct participants in your usability study can meaningfully optimize your sample size without affecting the results. That is why if you’re finding it difficult to recruit high-quality participants, we recommend tapping the help of research recruitment experts. By using a UX Research platform with a dedicated user panel, you can remove the hassle of finding the right participants by yourself. UXArmy has a large pool of high-quality participants you can use for your usability testing needs.  Visit uxarmy.com to learn more about UXArmy usability testing platform and our user panel known as the UserAdvocate community

Recent Post

Get The Notification Update News from Us

Share this Post