Planning a usability test
We always recommend user testing as the best way to find out what users think and do with your digital product in a controlled environment. In this post, we are taking a closer look at the planning and execution of these tests, following the Checklist for Planning Usability Studies developed by Norman Nielsen Group.
Norman Nielsen suggests a 7-step process, which we will describe deeper later:
- Define the objectives of the study.
- Determine the format of the study.
- Set a maximum number of users.
- Select your participants carefully.
- Plan tasks aligned with the objectives of the study.
- Design a pilot study.
- Write a Test Plan.
After explaining each point, we will tell you about our experience in Torresburriel Estudio, and why we believe that the framework proposed by NNG is very suitable for user testing.
Define the goals of the study
It is necessary to identify what the actual goals of the research are, as this will help you to select the methodology to use. Usability studies can collect either quantitative or qualitative data, so the different alternatives for each method need to be considered.
Reduce the goals of the interviews to an easy to manage number to be successful. If you need too many questions, users will be overwhelmed and your study will most likely fail.
Determine the format of the study
It is very important to determine the appropriate type of research for each project. Here are three questions you need to consider before developing the plan:
- Will the study be run in your lab or in a client’s facility?
- With or without a moderator?
- In-person or remote?
Some options will be more appropriate depending on the characteristics of the project. There is no perfect recipe and it depends on the environment of each studio.
Set a maximum number of users
Depending on the study, you will need a different amount of participants. Broadly speaking, if it is a qualitative study you can draw conclusions with 5 to 8 participants, but if it is a quantitative study you will need 20 to 30 participants for the results to be meaningful.
Select your participants carefully
It is very necessary to recruit participants who match the profiles of the actual users of the product. If they are asked to imagine scenarios to answer the questions, the validity of the study may be compromised.
Plan tasks aligned with the objectives of the study
We usually ask users to complete tasks while using an interface. Tasks are described as scenarios and should match the objectives of the study. Scenarios can be general or specific, and they can be of two types:
- Exploratory tasks. They are open-ended tasks and are intended to learn how people discover and explore information. This type is used for qualitative studies.
- Specific tasks. These tasks usually have a given correct answer.
Design a pilot study
It is very important to run a pilot study to make sure that everything is properly planned, in order to be able to solve any issue before starting the real interviews with all the participants.
Write a test plan
Ultimately, once you have thought about how the study will be conducted you have to create a document in which all the proposed information is collected. It should include information such as what is being tested, goals, logistics, participant profiles or metrics.
Our experience
At Torresburriel Estudio we have carried out many usability tests for different products and services, in a wide variety of sectors. If you are aware of your goals and the information you want to obtain from a test, it is easy to choose which type of testing to run.
Goals should be concrete and SMART (specific, measurable, achievable, realistic and time-bound). For example, if you want to analyse a menu, you can test information architecture using tree tests (if you are at an early stage of development) or you can do a user test with tasks (if you are at a more mature stage of development). Good information architecture is essential for a quality digital product.
Selecting the profiles is crucial, because you need to have users matching all of them (if there is more than one). If the user profiles are very broad and different, it is likely that you will need to make specific tests for each one.
If you test remotely, 10% of the users receiving your email will do the test, but only 3–5% will do the whole test. On the other hand, if you do them in person, keep in mind that some people will fail even if they say they will come, but success rate will range from 30 to 65%.
These data are no absolute truth, but the results we have had in our own tests. If the recruitment is done properly, rates will be higher, but people are often unwilling to participate in this type of study.
Designing the tasks correctly is key. In our view, this is one of the two key factors for a user test, being the other one recruitment. Explaining what the user has to do is critical, because if the users do not understand correctly what they have to perform, the test result may not be accurate. This is why running a pilot testing is so important.
Making questionnaires easy to analyse will make data collection easier. Taking notes during the session and recording the session (with the user’s consent) will also help to make a more complete repost to present to the client.
