Few vs several users do not determine whether a certain study is Qualitative or Quantitative. In case a quantitative usability study is required, it needs to be planned and designed with benefits over or in addition to a qualitative usability study. It is also important to clarify that A/B testing on Live websites is not a quantitative usability study. While it is good to be data driven, quantitative studies alone might not be the sole methodology to provide validated insights.
When to use quantitative usability study?
The need for quantitative Usability studies generally arises when the stakeholders have business critical questions to be answered. Since the results from quantitative usability studies provide measurements backed by desired level of statistical confidence, the subjective variations in qualitative usability studies are taken care of. Quantitative studies go by hard measurements, making it easier to relate to business metrics like Net Promoter Score (NPS), Return on Investment, correlations between usability problems and their effects on revenues, etc.
The quantitative studies help to statistically establish the existence or nonexistence of certain usability problems and relate them with certain business Key Performance Indicators (KPI). By using the findings from Quantitative studies one can eliminate skews, outliers from the findings and action on the usability findings with a desired degree of confidence. Like with any other quantitative study, there is a certain margin of error for each usability finding. However the results are statistically proven and quantity of error is known.
Example
A classic case where quantitative and qualitative studies are applied closely one after the other (not necessarily in that order) is the usability evaluation of an existing website. Such a website should be having thousands and thousands of daily visitors.
A qualitative usability study on this kind of a website does help to discover usability problems. The number of findings vary based on the attributes of researchers and participants involved. Due to the variations in knowledge, behavioural attributes and limited number of representative users, there will be subjectivity in the findings. It is common to find variations when certain variables are changed for instance, study conducted remotely or in-person, moderated or unmoderated, etc.
Among all usability issues discovered, a qualitative study will not be able to provide certainty of impact of each of the issues. It is possible that a majority of people in the target audience would just be able to live with few of those usability problems without any impact on engagement.
To measure the likely impact of each usability problem on the target audience of this website, larger sample sizes and more controlled conditions are needed. A quantitative evaluation is thus necessary. As an example of controlled conditions, if the participants are supposed to perform a Task (assume the Task flow has usability issues) in a quantitative study, all participants must do that Task in either unmoderated study or a moderated study, but not in a mix of both types. To add on, other variations like usability test tools, interpretation of Task descriptions/Survey questions, hardware (Computer vs Smartphone), internet speed, etc should be negligible within each of the user segments. The questions or the order of Tasks could be randomised to reduce bias.
The results from quantitative studies need to be analysed after breakdown into well defined user groups or segments. The impact of a usability problem to each user segment can be different. The quantum of impact can be found out via statistical analysis of the responses collected via the quantitative usability study.
When to use qualitative usability study?
For a new product development, several small qualitative usability studies are useful early during the design phase. These studies help to find and eliminate major usability problems swiftly and early in the product development iteratively. For situations needing a competition benchmark or a major redesign effort to improve user experience, quantitative studies must be added to the Usability evaluation plan.
Based on our experience at UXArmy conducting thousands of usability studies, an unmoderated remote usability testing can be performed as both qualitative and quantitative. Usability testing tools like UXArmy provide several quantitative measures for instance, time of task completion, task success / failure, number of clicks, pages/screens during navigation, response to survey questions (like multiple choice, checkboxes, Likert scale, etc). The screen-interaction in Videos along with think-aloud feedback and responses to open text survey questions should be analysed for qualitative insights.
Sometimes, it may be required to additionally conduct a dedicated survey (using a tool like SurveyMonkey, Qualtrics, etc.) to support the findings from usability studies.
I’m a product Designer. What type of usability study should I be doing?
There are situations when only doing the qualitative usability studies makes sense. “Learn early, often and fast” still remains an important mantra for developing usable products using usability testing. During design of the product, quick and inexpensive qualitative feedback is much more suited for use by the Designers. Quantitative usability evaluation is best suited to be done closer to product launch or after the launch. Qualitative usability studies would elicit users’ opinions and behavioural reasons behind ‘Why’ certain usability issues are problematic for the users. The complementary quantitative usability study shall measure and prioritise among the usability problems.
Product designers who are not skilled at statistics could collaboratively work with those researcher colleagues who may be able to help with statistical analysis of quantitative usability studies.
Conclusion
For usability-tracking of a product in a longer term, quantitative usability testing must be part of the Test plan. A common misconception is that qualitative study finds more problems than quantitative study. To get the most effective outcome from quantitative studies, controlling the variations as much as possible is of utmost importance . Like with any user research method, qualitative and quantitative usability studies have their respective strengths. UX Designers and researchers need to analyse the goals, consult the stakeholders and propose the most appropriate technique, or a mix of techniques, depending upon the benefits and available resources.