At UXArmy we have a rule for user research — do not proceed with a user research until the goals of that research have been fully established. Clear goals linked to business objectives are fundamental to derive the purpose of user research. The purpose is then further broken down into scenarios and associated questions that users would be asked during the research.
Between start of the recruitment and meeting the respondents, user researchers usually write down and refine the areas which need to be probed. A little more preparation by framing the right questions, leads to a better research outcome.
At UXArmy, we ask ourselves these two questions at the start of a research project:
Based on the above, the research questions are prepared. After the research questions are ready, we critically look at the following:
The process of selecting questions for moderated and defining tasks for unmoderated online user research is similar. Depending on the type of research (Qualitative / Quantitative) being conducted, the type and style of asking questions varies. Qualitative user research is used for discovery of “Why” and “How” whereas Quantitative research is more about the “What”, “Where” “When” and “Who”. In most cases, the outcome of Qualitative research is treated as ‘Hypothesis” which is then validated using Quantitative Research, but this order can be interchanged depending upon the project goals.
At UXArmy we apply several methods of qualitative user research, the choice of a particular method is impacted by the research objective. Some of the most frequently used methods are ethnographics research, diary studies, focus groups, in-depth interviews and sometimes unmoderated remote user testing.
In this blog, I would touch upon the guidelines for framing research questions for face-to-face, in-depth interview method of qualitative research, when applied to study of website or mobile apps. To keep a context to these ‘question formulation’ tips below, let’s assume the goal of this user research is discovering problems and opportunities (for new business) in a mobile app.
We make sure to avoid any question that could be answers as Yes or No, Agree or Disagree, etc. Open ended questions are likely to reveal much more from the respondents, upon which more probing questions can be built upon.
For instance, asking
“Do you know about features of this App?”
It is actually better asking
“Describe the features of this App.”
Based on the description of features provided by the respondent, pick up cues on likes or dislikes for specific features. Those cues would help to build further questions that need to be asked in relevance to the purpose of the user study.
Qualitative research is focussed on finding more about the subjective aspects of user behaviour. Based on verbal response and behavioural observation, probing questions need to be built.
For instance,
“I noted that you chose to dismiss that screen three times, why was that?”
..or building upon any statements from the respondent would help to uncover reasoning behind specific behaviour / style of usage:
For example, if the respondent had said
“I can’t imagine browsing by swiping the images horizontally ‘left to right’ or ‘right to left’, vertical scrolling is better”, the following probing question can help to investigate deeper in to the reasons.
“Would you explain that further?”
Probing questions also include — asking for an example, to describe an applicable situation, or simply
“Is there anything else?” or, “Can you explain that again?”
Think about a situation where a respondent is using a longer navigation path to access their photo library. A new researcher could be asking this:
“Why do you think you would use that screen more frequently to access your picture library and not the other method which is present in the App”
…the respondent may even forget such a long question altogether.
The above question can be framed in a more simple way as multiple level of questions:
Level 1: Can you tell why you access the picture library using this ScreenA?
In case the respondent does not talk about the other ways to access the picture libraries, further questions can be asked:
Level 2: “How about using ScreenB to access the picture library?”
Followed by a question to compare the three ways to access the photo library
Level 3: “Can you tell me the reasons of using ScreenA over ScreenB?”
This one is hardest for new researchers. As humans we have our biases and on top of that our brains are exceptionally smart to create new bias instantly. The same can happen during the interview session.
Imagine if a respondent says
“I actually missed the “Laundry” button on the Home page.”
A new user researcher might be tempted to ask a prompting question…
“Did you miss it because of the light colours or position of the button?”
,,,,simply because the previous respondents might have missed seeing the button due to colour or placement of the button.
Whereas, an experienced researcher would always ask
“Why did you miss the Laundry button?”
….that question allows the respondent to think through and express the actual reason of missing the button. The reason could well turn out to be the choice of laundry stock image which, overpowers the subtle visual design of the button.
Besides using abbreviations, usage of internal terminology is strict no-no!
While you might be using the terms everyday in internal discussions with stakeholders, it means nothing to the respondent.
I have have seen young researchers occasionally using the very commonly abbreviations like “FYI”, “KIV”, “ASAP”, etc. without realizing that these might sound completely Greek to your respondent. Avoid them altogether.
Few more examples of bad questions:
“How did you shop for during the CNY Sale?”
After hearing this question, the respondent was looking blurred, because he had shifted to Asia only 6 months back. He is not know CNY stands for Chinese New Year.
“How did you feel about using the COD payment option?”
In this case also, the respondent had no clue of what COD payment means. Cash on Delivery (CoD) payment options are not an online shopping payment method in only all countries.
While interviewing in a common foreign language (for example English) completely avoid complex words of which the respondent might not know the meaning itself. Remember that the respondent might be coming from a cultural background in which using English for day to day conversations is not preferred.
The Takeaway
Above are the five most important ‘question formulation’ tips that could take time to master. A highly skilled user researcher would ask the right questions in the right way because she assumes nothing and optimises her communication with the respondent. It takes experience to be able to apply empathy and an array of other research skills into practice — there is no fast track substitute to experience. Mark Twain said, “To a man with a hammer, everything looks like a nail.” — to do an effective user research, multiple skills like empathy, neutrality, communication, persuasiveness, collaboration, etc. are needed. User researcher must develop all the required skills over time.