Ten myths about UX metrics
There are many myths about UX metrics. Some come from a lack of experience in using them, others from a negative experience with them, and others from complaints from some UX professionals about the problems and costs associated with their use. The source of these myths is not important. In the end, what matters is separating the facts from those that are not.
If you are interested in learning UX Research don’t miss our UX Learn trainings (in Spanish):
So here we have listed the ten myths about UX metrics that we have encountered on our journey with the discipline.
Myth 1: getting metrics takes too much time
At best, metrics can speed up the design process and, at worst, they should not influence overall times. Metrics are collected quickly and easily as part of a normal iterative usability evaluation. People on the team may wrongly think that full surveys or UX lab tests have to be done after basic metrics have been collected.
But there are some fairly simple metrics that can be collected on a daily basis. Adding some questions at the beginning or end of each usability test session does not affect session time and can provide useful information. Participants can quickly respond to some key questions as part of the classic background questionnaire or follow-up activity.
Participants can also rate activities for ease of use or provide a satisfaction questionnaire after each activity or after all tasks have been completed. If access to a large user group is easily available, it is possible to send a mass email with some key questions, adding some screenshots. This way, data can be collected from hundreds of users in a single day. Some data can be collected quickly without involving the user.
For example, it is possible to quickly and easily indicate the frequency or severity of specific problems with each new iteration of the project. The time required to collect metrics does not have to be weeks or days. Many times only a few hours are necessary.
Myth 2: metrics are too expensive
Choosing the right metrics in UX is key to getting reliable results. Only in this way will it be understood how its use is an investment and not a cost or worse, a waste.
Myth 3: metrics are not useful when focusing on small iterations
Some team members may question the usefulness of metrics when they are only interested in small iterations. They may say that it is better to focus on a series of small improvements and not worry about metrics. They may not have time or budget to collect metrics and so may say that metrics do not fit into the iterative design process.
Analyzing the usability of a project to determine the severity and frequency of usability problems is essential for all projects. This is an excellent way to focus resources during the planning process and will save time and money. In this case, we can easily obtain metrics based on previous studies that can help us respond to problems more quickly and effectively. Metrics are ultimately useful for both large and small projects.
Myth 4: Metrics do not help understand the cause of the problem
Some people claim that metrics do not help understand the main cause of user experience related problems. They think that metrics only serve to highlight the extent of the problem but not the cause. If we focus on successful cases or data, it is easy to understand why some people have this position.
Metrics can tell us a lot more about the main causes of usability problems than we initially think. It is possible to analyze user feedback to reveal the source of the problem, it is possible to identify the exact situations in which users encounter a problem and use metrics to indicate where and why some problems occur.
Depending on how the data is coded and the methods used, there is a large amount of data that can reveal the main cause of many usability problems.
Myth 5: Metrics are full of contaminated data
One of the main criticisms of metrics is that the data is contaminated. Too many variables prevent a clear view of what is happening. Let’s look at an example of contaminated data. When we measure task completion time in a usability study and the participant goes for a coffee, how much does it influence the assessment of task time? Although it happens occasionally, it should not prevent us from collecting data on task times or other usability data.
There are things that can’t be done to minimize or remove contamination in the data. Data can be “cleaned” so that extreme values are not used for analysis. In addition, it is possible to carefully choose specific metrics to mitigate “contaminated” data.
It is possible to use well-defined procedures to ensure appropriate levels of consistency in task assessment or usability problems. Many standard questionnaires have already been widely validated by many researchers to avoid exactly this type of problem.
Myth 6: Relying on intuition
Many usability decisions are based on intuition. There is always someone on the team who thinks that the decisions being made are right or wrong based on intuition. The beauty of metrics is that getting usability test data requires a lot of reasoning, some design options are truly edge cases that could have a strong impact on the project. Sometimes that happens: the correct design solutions are just the least intuitive.
For example, a design team can ensure that all information on a website is “Above the Fold”, thereby eliminating the need for scrolling. However, usability data (for example in the form of times to complete a task) may show that scrolling is actually easier for users and that above the fold designs are not ideal.
To avoid these situations, it is essential to use metrics to support usability decisions and to be aware of the limitations and potential pitfalls of relying solely on intuition.
Myth 7: Metrics cannot be applied to new products
Sometimes metrics are not used when evaluating a new product. Some people hold that, since there are no products to compare, metrics have no sense. In reality, it is exactly the opposite.
When evaluating a new product, it is essential to establish a series of reference parameters against which to compare future design iterations. It is the only way to truly know if the design is improving or not.
In addition, it is useful to establish reference metrics for new products. Before a product goes into production, it must meet base metrics related to success, satisfaction, and efficiency.
Myth 8: There are no metrics for the type of problem you are facing
Some people believe that there are no metrics specifically related to the product or project they are working on. Whatever the goal, at least a couple of parameters should be directly tied to the product’s business objectives.
For example, it may be said that interest is only in the emotional response of users and not performance. In this case, there are also different established methods for measuring emotional responses. In other situations, someone might be interested only in “awareness.” There are also simple ways to measure “awareness” without investing in an eye-tracking study setup. Others say they are only interested in users’ more subtle reactions, such as frustration level. There are methods for measuring stress level without directly asking users.
Myth 9: Executive management does not understand metrics
Although some executives view User Research as only qualitative feedback about a product, fortunately, most executives recognize the value of research. Metrics give credibility to the team, product, and design process.
They can be used to calculate ROI (read also about Usability ROI calculators) and most executives love that data. Metrics can be critical in opening eyes to business issues. It’s not the same to say there’s a problem with the online payment process than to say x% of users are unable to successfully complete the purchase.
Myth 10: It is difficult to gather reliable data if the sample is small
There is a widespread idea that to gather any metric, a large sample is required. Some believe that to collect data and talk about metrics, a minimum of 30 participants must have been tested. This is not entirely true. Having a wide sample dimension can help improve confidence level, but there are tests that can be perfectly performed with small samples like five users, such as qualitative studies that aim to identify usability problems with an interface.
As UX experts we have to keep working to improve the UX maturity of organisations and get the people involved to understand that we have to rely on UX metrics and make data-driven decisions.
This is a translation of the following article from our corporate website:
