21 methods of UX research: when to use which
Like it or not, there’s no way of getting around the topic of UX testing when creating a digital product. Any specialist who gives even the slightest damn about their work wants to know that the results of their hard spent man-hours will at least be appreciated by the end users.
Some of the methods of UX research are so obvious and straightforward that when IT specialists are first acquainted with the classification, a lot of them ask the logical question: “Do people actually avoid doing this?” This question is usually followed by “Have they been fired yet?”
Let’s face it — we all test the usability of our products, even if we don’t explicitly set out to do so. Some use the pretentious nomenclature “research study”, while others don’t call it anything at all, but they still test their products on the down-low.
This article is a compilation of twenty one UX research methods. Some of them are simple and commonplace, while others are a bit more sophisticated and complex. Choose the one that suits your taste and your purpose, but first, let’s take a look at what characteristics set them apart from each other.
Differences in the Methodologies
Qualitative / Quantitative. A part of the research study can provide some sort of measurable (quantifiable) result. For example, data that is collected by an analytics system or information derived from the results of a survey. This type of research is described as quantitative. On the flip side, in some types of research, only a qualitative assessment is possible, i.e. the results consist of a subjective interpretation of the researcher or participant (convenient vs inconvenient, simple vs confusing, etc.).
This is logically related to the number of participants in the study. The more of them there are (quantitative method), the more difficult it becomes to analyze each case individually and, consequently, the easier it is to operate with average numbers. The fewer the number of participants (qualitative method), the less sense it makes to rely on the numbers, since the results can vary greatly in small samples. In this case, one has to resort to a qualitative assessment and find out “why” someone gave a certain answer or did a certain thing, instead of “how many” people did so.
Behavioral / Attitudinal. Behavioral research methods tend to answer the question “What do users do?” While attitudinal methods focus on “What do users think?” The answers don’t always mesh, but depending on the purpose of the study, both are equally important.
Context of Product Use. The following methods establish the degree of the product’s involvement in the research. The study can be conducted:
- in conjunction with the natural use of the product
- in conjunction with a pre-created scenario (script) for using the product
- without the product’s involvement in the study
- with mixed use of the product, or a combination of the above mentioned methods
Phases of Product development. Let’s highlight three phases of a product’s development cycle: strategy, execution and assessment. Research methodologies will vary depending on which of these phases the product is in at the time of performing the study.
We’ll start with the methods that are used early on and conclude with the methods that are performed at the end of the product’s development cycle.
UX Research Methods
1. Concept Testing
2. Card Sorting
3. Focus Groups
4. Ethnographic Field Studies
5. Participatory Design
6. Tree Testing
7. Desirability studies
8. Usability-Lab Studies
9. Eyetracking
10. Diary/Camera Studies
11. Moderated Remote Usability Studies
12. Five second test
13. Unmoderated Remote Panel Studies
14. A/B testing
15. Usability benchmarking
16. Expert review
17. Clickstream analysis
18. Customer Feedback
19. Email surveys
20. True intent studies
21. Interviews
1. Concept Testing
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Mixed
Project Phase: Strategy
Overview
This method’s focus is to highlight the key qualities of the product in order to determine if it meets the needs of the target audience. Anyone who is familiar with the MVP (minimum viable product) concept will likely understand what this method is all about. It can be performed in either the one-on-one format, or with a large audience. The main goal of this method is to understand if there’s even a need for the product in question.
When to use
This research method is generally used whenever there’s a need to confirm the viability of a concept before developing it into a fully-fledged product.
2. Card Sorting
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Natural
Project Phase: Strategy
Overview
It can be said that this method is the minimalistic version of getting the user to help design your product. This involves making a draft of the product’s information architecture. Users then receive sets of cards that correspond to the data that can be found on the site or in the application.
Guided by their own logic, the users then divide the cards into semantic groups, which can then become the foundation for future interface screens. Users are often expected to come up with the group names in this method.
When to use
Card sorting can be used in the early phases of product design, especially if the product doesn’t have any direct competitors on the market. It can also be used to detect errors in an existing information framework.
3. Focus Groups
Qualitative / Quantitative: Qualitative
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Not used
Project Phase: Strategy, Execution
Overview
A group of 3–10 participants, which is led by a moderator, discusses their views on a future product. The role of the moderator is to maintain the flow of opinions rather than directing it. A focus group can answer a few basic questions, but the session shouldn’t turn into an interview (see method #21).
When to use
Focus groups are best implemented at the very beginning of the project, in order to understand what users expect from the final result. This method is useful for making important strategic decisions and forming the product’s general concept.
4. Ethnographic Research Studies
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Either/Both are possible
Context of Product Use: Natural
Project Phase: Strategy, Execution
Overview
Ethnographic research studies are conducted with the use of an existing (competing) product on the market, which is similar in concept to what you want to create yourself.
With this method, your goal is to “catch” the user in their natural environment. This includes observations like monitoring a company’s employees while they’re using a CRM system in day-to-day work.
Ethnographic studies assume the physical presence of a researcher. The studies themselves focus more on how people interact with each other rather than on how they use the product. This research method requires a great deal of preparation and may include passive observation, participatory observation, and contextual interviews (questions during the work process).
When to use
Ethnography is used to monitor similar, existing products when your own product is in its early phases of development. This method provides critical insights into users’ needs, which allows you to refine your overall product concept before it goes to market.
5. Participatory Design
Qualitative / Quantitative: Qualitative
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Natural or Scripted
Project Phase: Strategy, Execution
Overview
Users are asked to independently draft the structure of the application on a piece of paper or a whiteboard. While knowing the product’s purpose, they try to present it in a form that would be most convenient for them.
To facilitate the process, cards that represent functional elements, decorative materials and markers are used as the materials to create the draft. The users, in this scenario, can be represented by the target audience or the client’s employees, managers, shareholders, etc.
When to use
This method can be used in place of a product design brainstorming session. It allows you to find unexpected, useful ideas that can help improve the users’ experience when they’re working with the solution.
6. Tree Testing
Qualitative / Quantitative: Quantitative
Behavioral / Attitudinal: Behavioral
Context of Product Use: Not used
Project Phase: Strategy, Execution
Overview
The study’s participants work with a text-only version of the site or application, the entire structure of which is presented in the form of a tree. Top-level categories expand into sub-categories, which also expand into sub-categories, and so on.
The task for the participants is to find a particular menu item or site element using only the presented tree diagram. The result is measured by the distribution of clicks in each category. This method often shows how misleading product structures can be.
When to use
Tree testing can be used to detect possible navigational difficulties, as well as to better understand users’ logic.
7. Desirability Studies
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Natural, Scripted
Project Phase: Strategy, Execution
Overview
Participants are shown several design options. They must evaluate each option with a set of quantitative and qualitative characteristics, such as: speed, simplicity, usefulness, predictability, freshness, authority, etc. The characteristics are provided by the researchers in advance, and often cards are used for the sake of convenience.
When to use
This research method is used when you need to understand what kinds of subjective feelings the design evokes in users. It’s advisable to perform this research method in the early phases of the product’s development cycle.
8. Usability-Lab Studies
Qualitative / Quantitative: Qualitative
Behavioral / Attitudinal: Behavioral
Context of Product Use: Scripted
Project Phase: Execution
Overview
Participants perform suggested tasks using the product under the supervision of a researcher. To do this, users are provided with a scenario of the necessary actions with an explanation of their goals. Important details of the various operations (order, time elapsed, etc.) are recorded by the researcher.
You can read more about how this research method is carried out in the article “UX-research of Remote Banking Services: our experience, mistakes and discoveries”.
When to use
This method can be used during the development of a UX solution, in order to validate or refute various hypotheses. Lab studies can be performed as early as the prototyping stage. They are suitable for proprietary applications that are not available to the most users.
9. Eyetracking
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Behavioral
Context of Product Use: Natural or Scripted
Project Phase: Execution
Overview
A special device, called an eye-tracker, measures the fixation points of the user’s gaze when working with a website or application, as well as the transitions between these points. To establish gaze points during the study, an infrared eye-tracker fixed on the monitor is often used to record reflections from the user’s eyes.
This research method shows which interface elements receive the most attention from the user. It also allows researchers to discover areas that are confusing to users (for example, complicated menus where the user fails to find a desired element). Eye-tracking studies can involve scripted instructions or be conducted without the use of prepared instructions.
When to use
This study is best performed during the execution phase of a product’s development, when a usable version is already available. It is most commonly used to optimize the structure and navigation of the interface.
10. Diary / Camera Studies
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Natural
Project Phase: Execution
Overview
Participants in the study regularly record information about the cases in which they think about your product. In order to get a broader picture, they write down the actions, events, the situations that led up to such thoughts.
Information can be recorded in various ways, such as: with a paper journal, a note-taking mobile app, or audio and video messages. Users are instructed on the format for note-taking and, if possible, are asked to set up daily reminders.
When to use
This research method is useful when you need to better understand the context in which the user is likely to encounter your product. It also helps to understand the users’ motivations and their daily habits.
11. Remote Usability Studies
Qualitative / Quantitative: More qualitative than quantitative
Behavioral / Attitudinal: Behavioral
Context of Product Use: Scripted
Project Phase: Execution, Assessment
Overview
A counterpart of laboratory research studies that are carried out remotely, for example, with the help of remote access software. Sometimes these studies are moderated, when participants are asked to respond to comments and questions from a specialist. Or they can be unmoderated, when tasks are performed independently by the participants.
When to use
These studies are used in the same circumstances as laboratory research, except they require fewer resources. Remote studies allow you to invite a greater number of participants and get more realistic results due to the fact that the participants use the product in a more comfortable setting.
12. The Five Second Test
Qualitative / Quantitative: Qualitative
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Not used
Project Phase: Execution, Assessment
Overview
This is a sort of “lightning round” test to gauge users’ first impressions of the product’s design. The study’s participants are shown a fragment of the product for exactly 5 seconds, after which they answer a series of questions. Some examples of questions include recalling the main elements of what they were shown, what impression they developed about the brand, what the purpose of the shown page was, what they think the target audience is, and so on.
When to use
The five second test is used whenever you need to make sure that the designers’ vision of the product matches the vision of the users. If the discrepancy is noticeable, it’s better to take the time to correct the concept.
13. Unmoderated Remote Panel Studies
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Either/Both are possible
Context of Product Use: Natural or Scripted
Project Phase: Execution, Assessment
Overview
A panel is a specially trained group of users who use the product independently and “think out loud”. To accomplish this, video recording applications are used during the study. Consider this a more simplistic alternative to ethnographic research.
The pros of this approach: saved time, the ability to test more actions, and a fresh perspective. Cons: difficulties in analyzing the results.
When to use
This research method is best suited for studying products that have long use cycles (fitness apps, health monitoring programs, etc.). It can also be useful for analyzing pending events (such as notifications) and to plan product updates.
14. A/B Testing
Qualitative / Quantitative: Quantitative
Behavioral / Attitudinal: Behavioral
Context of Product Use: Natural
Project Phase: Execution, Assessment
Overview
This is a popular method for comparing two versions of a site or application that differ in one or several elements. The audience is randomly split into two segments. Each segment only uses one of the two versions.
After reaching statistical significance, the results are used to conclude which option was the best according to the selected KPI (for example, in-app purchases). A/B tests are performed by using specialized services, such as Google Optimize for websites, and Optimizely for apps.
When to use
A/B testing can be used to optimize a working version of the product, that is to say, either at the last phases of development, or after the product’s release. These studies help “fine-tune” critical interface elements such as CTA buttons or navigation elements.
15. Usability Benchmarking
Qualitative / Quantitative: Quantitative
Behavioral / Attitudinal: Behavioral
Context of Product Use: Scripted
Project Phase: Execution, Assessment
Overview
This is a laboratory research method that’s conducted regularly in order to track changes in usability. The methodology is comparable to the previous one, but the main difference lies in how the results are compared. Benchmarking can be used to compare the performance with previous versions of the product or with its direct market competitors.
When to use
Whenever there’s a need to track or demonstrate progress in the product’s optimization. Benchmarking can also be used to evaluate the effectiveness of the solution as such, especially when switching between development cycles.
16. Expert Review
Qualitative / Quantitative: Qualitative
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Natural or Scripted
Project Phase: Execution, Assessment
Overview
The product is peer reviewed by an experienced UX specialist, who documents their observations into a detailed report that is supplemented by illustrations. This document describes the strengths and weaknesses of the solution, identifies the problems and their causes, and lists recommendations for the problems’ elimination as well as the best practices for doing so.
When to use
This method is often implemented in order to get a fresh outside perspective on the product. It also helps highlight the little things that the untrained eye doesn’t notice. This research method is also suitable when the UX design team lacks the expertise necessary to “polish” the product.
17. Clickstream Analysis
Qualitative / Quantitative: Quantitative
Behavioral / Attitudinal: Behavioral
Context of Product Use: Natural
Project Phase: Execution, Assessment
Overview
This research method involves analyzing data that shows which pages the user visited and in what order. It’s quite easy to conduct this type of research with the help of systems such as Google Analytics, Yandex.Metrika, Firebase, or Mixpanel. This analysis allows you to identify problems related to the navigation of a site or app. However, it doesn’t help to determine their causes. If you want to find out the reasons why something isn’t working too well, then you should consider trying usability research methods #8 and #11.
When to use
Clickstream analysis is useful if you want to check whether a product is being used as intended. This research method can be performed on the final or intermediate version of the product.
18. Customer Feedback
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Natural
Project Phase: Assessment
Overview
A product experience questionnaire is sent to a random sample of users. There are many ways to accomplish this: from web forms and pop-ups to polling services and email surveys. You can combine closed and open-ended questions, so long as you don’t overload the respondents.
When to use
This research method is best implemented after the user performs a meaningful action on a website or app, such as placing an order or requesting support. Launching a survey earlier can complicate the path to conversion and alienate the visitor.
19. Email Surveys
Qualitative / Quantitative: Either/Both are possible
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Not used
Project Phase: Assessment
Overview
As opposed to collecting feedback from users, email surveys are general and not triggered by actions. The surveys generally inquire about a user’s previous interactions with the product, so they can only evaluate their perception of it. This research method is easier to conduct than interviews (see method #21), but is more likely to be refused by potential participants. The number of questions usually doesn’t exceed 10, so as not to scare away or burn out potential respondents.
When to use
Email surveys are used to analyze the effectiveness of an existing product and compare it with competitors.
20. True Intent Studies
Qualitative / Quantitative: Quantitative
Behavioral / Attitudinal: Either/Both are possible
Context of Product Use: Natural
Project Phase: Assessment
Overview
This study involves giving a survey to random users of the site / application before and during its use. Upon opening the site / app, users are asked about the purpose of their visit and then after some time, whether they managed to accomplish their goal.
Sometimes the survey also asks for demographic information or other questions that are relevant to the study. It’s important not to overload the users with questions, or else the visitors will have no desire to fill it out.
When to use
True intent studies are best implemented after the product has already been launched, and when a sufficient level of traffic is reached. Its purpose is to find out if the developed product meets the needs of the users, as well as whether or not it fulfills its intended purpose.
21. Interviews
Qualitative / Quantitative: Qualitative
Behavioral / Attitudinal: Attitudinal
Context of Product Use: Not used
Project Phase: Assessment
Overview
This study involves conducting personal interviews with people who have experience with the product. The goal of the study is to understand what impression the site or application leaves on its users. Since the interview gathers information about the past, it will not be possible to obtain accurate numerical data.
When to use
Interviews are useful to perform when planning a redesign or upgrade of an existing solution. They allow you to assess the users’ perception of the product and the brand, as well as highlight the primary technical weaknesses.
Conclusion
As you can see, there is a whole arsenal of methods for analyzing and identifying the reasons for users’ behavior and their perception of the product. Each approach can play a critical role in creating a UX solution if you know when to apply it (or it may not). Hope this guide was helpful.
Made by opium.pro. Writers: Denis Elianovsky and Stanislav Lushin. Thanks to Elena Efimova for the infographics, Tatiana Kitaeva for the editing, and Pavel Chernetsov for the English translation.
