Planning usability studies the right way
At Torresburriel Estudio, we recommend user testing as the best way to check what users think and do with your digital product, in a controlled environment. Today we want to delve into these tests and some guidelines for their planning and execution, which Norman Nielsen Group has collected in his article “Checklist for Planning Usability Studies”.
In this article, you will find an 8-step process:
- Define the objectives of the study
- Determine the format and setup of the study
- Determine the maximum number of users
- Invest time recruiting the right participants
- Design tasks aligned with the study objectives
- Conduct a pilot study
- Define your metrics to get the right insights
- Document your test plan
At the end we will tell you about our own experience and why we believe that this framework proposed by Norman Nielsen Group is very suitable for conducting user tests.
Define the objectives
A meeting with stakeholders will help to determine what they are looking to find out. Here, you will be able to identify the questions, concerns, areas of interest, and purpose of the investigation. The objectives will determine the research method used.
Usability studies are great to collect qualitative or quantitative data on behavior and to solve questions related to design. For example: is the content presented in a way that is easy to find and understand? Can the user finish a task successfully? If your goal is to collect data on attitudes, consider alternative methods that are more suitable for those goals.
By the way, precision at defining objectives is a must, as each additional question will reduce the quality of the perceptions of the rest of the objectives of the test: focus on the research goals that truly affect the ROI.
Format and setup
Some considerations we have to make to determine the type of investigation appropriate for your situation:
- Lab or field studies: Should the study be carried out in your own facilities or those of the participants? For convenience, most of the face-to-face usability tests are done in-house or in laboratories. However, if the environment is critical or difficult to emulate, the scrolling may be worth it.
- Moderated or unmoderated: Moderated studies tend to provide insights closer to the design and opportunities to test and ask to clarify them. They are also a better source of open comments from participants. On the other hand, unmoderated studies are cheaper, faster and can provide us with better access to difficult-to-recruit participants.
- In-person or remotely: our recommendation will always be to conduct in-person studies, when possible. When you are in the same room as the participant, the interaction feels friendlier and you are able to detect latent signals, such as body language. However, sometimes testing in person is not possible — because of budget, an Agile environment or because users do not come to you. And, of course, for public health reasons.
Maximum number of users
For traditional qualitative studies, we generally recommend 5–8 participants for the best ROI. If your study has more than one user segment, then you may have to ensure the number of participants per segment.
Quantitative or eye-tracking studies require longer samples to obtain meaningful conclusions. Calculate multiplying the number of participants by at least 4. You will need between 20–30 participants for each user group segment.
Spend some time recruiting the right participants
To conduct tests successfully you will need to get representative participants. The best insights come from real users’ feedback, especially if they match your target and, better, your personas.
Also, for B2B sites, you should find people who fit your exact circumstances.
Task definition
When running tests, ask users to complete tasks while using the interface. These tasks must be written in the form of scenarios and have to match the objectives of the study. The scenarios can be general or specific, and normally come in two main types:
- Exploratory Tasks: These are open response tasks, oriented towards objectives and may or may not have a correct answer. These tasks are intended to learn how people discover / explore information and are not appropriate for a quantitative study.
Example: you are interested in booking a hotel for your family and pets, see if the site has a suitable offer. - Specific Tasks: These are more focused and usually have a path or an ending, and are suitable for both quantitative and qualitative studies.
Example: Find the opening hours for a specific day of a library.
Take some time to define your tasks properly, as this point is critical to get valid results in your research. A solid task must be concrete and without clues that can optimize user behavior. Tip: if the same word appears on the interface, there’s a risk the activity becomes a word search game.
On the other hand, if the instructions are too general or vague, this can cause users to evaluate aspects that are not relevant to the study.
Conduct a pilot study
After you’ve written your tasks, be sure to conduct a pilot study. This will help you fine-tune your testing: tasks that can be done per session and its order. Pilot studies can help you also refine your recruitment criteria, so you are testing with the correct participants. It’s easier to solve any potential issue before the sessions.
Also, pilot studies are critical for unmoderated remote testing, as you won’t be able to clarify or make corrections if participants misinterpret instructions or task descriptions.
Metrics, metrics, metrics
The main purpose of a qualitative usability study is to gain insights about the design although, with few users, the metrics that may not be representative of user target.
However, in a quantitative study or with well-defined tasks and a large number of users, this step is important. The usual usability metrics are: time to perform the task, satisfaction ranking and success / error ratios.
If you decide to collect subjective information, such as ease of use or satisfaction questions about the task or the system, decide when you will deliver the questionnaires: after each task, at the end of the session, or both.
Document
Once you know how to conduct the test, document your approach and share it with your team. This document will help team members to understand how the testing will work and, also, will save time for planning future studies.
Even though the format is not so important, this document has to have some key information such as:
- Name of the product or site to be tested
- Objectives
- Logistics: times, dates, location and format of the study.
- Recruited participant profiles
- Tasks
- Metrics, questionnaires
- System description (mobile, desktop, computer options)
- Any other relevant information for the project
Motivate your fellow coworkers
A great benefit of usability testing is promoting collaboration and decision making as a team, especially witnessing how users respond to interface design.
Letting stakeholders observe moderated sessions reduces the amount of time required for communication and document findings. This helps teams spend less time guessing and arguing, and more time designing.
Tip: Invite your stakeholders and team members and give them a wide variety of reasons to come and participate.
Conducting the study in a traditional usability lab or a simplified one is ideal, but if your team members are in different locations, offer to view it remotely to make the activity inclusive.
Some notes about our experience
At Torresburriel Estudio we have conducted tests with users for different products and services, in a plenty of sectors and with both general and specific profiles for a certain group of clients.
The objectives are very important, so, if they are correctly defined as well as the results you want to obtain from a test, you can easily choose your testing technique.
We recommend you to check this infographics by Norman Nielsen Group and choose the most suitable for your research based on the type of data (behavior or attitude) that you want to analyze and whether you want quantitative or qualitative results.
The more specific the objectives are, the better. If you want to analyze a menu, do information architecture tests such as a tree test if you are in an early stage of development or a test with users with tasks using an alpha version if you are in a more mature stage of development. A good information architecture is essential to have a quality digital product.
Also, recruiting the right profiles and sampling is essential, because to success, you need users of all profiles if there are more than one, try that the proportion of each profile is similar to the proportion of users of each profile of your product, unless that you want the test is aimed at improving it for a specific profile.
Regarding remote or in-person tests, for remote testing, 10% of the users you send an email will do the test and they will complete it between 3–5% of the total, that is say between half and a third of those who open the test.
For in-person testing, keep in mind that many people will fail even if they have confirmed (keep this in mind so as not to get frustrated when doing them), but you will have a success of 1–2 out of 3 that they tell you that they are going to go. And, of course, respect health restrictions if applicable.
These are the data that we have had in our tests: if recruitment is done properly, there may be cases of higher percentages but people are usually reluctant to participate in this type of study, even if they get a reward or a gift.
It is our experience, so that you do not be desperate if your success rates seem not to be as high as expected.
Also, follow strictly Norman Nielsen’s advice on the number of participants, and take it as an actual rule.
Writing the tasks correctly is key, it is one of the two essential tasks (the other is a good recruitment) of user testing. Getting the user to know what he has to do is essential, because if he does not understand it correctly, test results may be invalid.
The metrics that Norman Nielsen says are the best but we would recommend that you also make participant observation and notes on non-verbal attitudes of the users, because most of them will not criticize the interface out loud but they will do gestures when doing the tasks. Many users when observed are cut off in their verbal reactions, that you must take into account and also that they can lie to you when answering the questionnaires, whether they are written or in person.
Doing some easy-to-tabulate questionnaires will do a lot of the work of collecting data to present a clear report. Writing down data about the user during the session or taking photos of the session (with the user’s consent of course), will also help to make a pleasant report to present to the client.
Communicating the results of a test with usability users is essential, as well as inviting the stakeholders to observe how these tests are carried out and to see in situ the reactions to their product, both things will help us to make the test results credible.
This article is a translation of the following one published on our corporate website:
