Book Review: Practical User Research: Everything You Need to Know to…” by Dr.Emmanuelle Savarit

“The issue with startups is not that they do not understand what UX is. I think they are just not ready for it”

“Stakeholders may not realize that face-to-face interviews with 12 people are more reliable than a survey with 100 to 200 respondents”

Have you ever wondered when you should involve user research into your product development? Or think that quantitative research is more reliable than qualitative research? Or wonder how user research can influence business strategy? Or confused about the difference between a user profile and user persona?

This book “Practical User Research: Everything You Need to Know to Integrate User Research to Your Product Development” authored by industry veteran Dr.Emmanuelle Savarit offers a systematic approach to incorporate user research into products. This book is a great read, and can be quite easily used in real life product management situations. Let’s dive in

Structure of the Book

The book is divided into 10 chapters. First 2 chapters concentrate on explaining to readers how user research as an expertise has come about and also how traditionally product development happens across industries. In Chapter 3, the author explains how product managers can introduce or evolve user research function into their organisation depending on their maturity. The author argues that the main obstacle to introducing user research into product development is maturity phase of the business. Chapter 4 and 5 explain how to prepare for user research

Chapter 6 explains main methods for quantitative research namely analytics, surveys and card sorting. Chapters 7 and 8 focus on collecting data and analysing findings for a few qualitative methods. Chapter 9 goes into aspects like ethics, compliance and consent in user research – areas which are generally considered as an afterthought. In the last chapter 10, author briefly touches on how user research projects unfold in real life through 10 case studies she has conducted across multiple industries and cultures

A. Introduction to User Research

User research consists of putting an end user’s needs at the center of the researcher’s investigation. Role of the user researcher is to provide evidence-based findings using quantitative and qualitative research methods. In user research, we are not interested in the demographics, job role, etc. We are interested in participant’s motivation or their behavior while using the product

Like many other disciplines, it traces its origins to the military. After WW2, there was a need felt for optimising weapons in order to minimise injuries to soldiers. Many disciplines like ergonomics, design, human-computer interaction, usability and user research emerged consequently

While doing research, we have the choice of taking a quantitative approach or a qualitative one. Often, stakeholders prefer numbers to case studies. However, when the research is properly done, both approaches are reliable and robust

Quantitative Research

• Takes a top-down approach, starting with the big picture and using deductive reasoning
  
• Validates a hypothesis, theory, or preconceived idea
  
• Answers the questions how many, how often, and when
  
• Doesn’t answer why or how people are using the website
  

Qualitative Research

• Takes a bottom-up approach and starts from a specific observation and goes to a generalization and informs theory. It for the same reason leaves room for unexpected avenues
  
• Identifies phenomenon, common patterns, and systematic occurrence
  
• Useful to answer the questions how and why
  

Often, businesses have difficulties understanding that user research is not going to provide them with what they want to hear but rather help them identify what the user needs to perform a transaction, to complete a task, or to carry out their daily job

A typical digital product development goes through 5 phases

1. Discovery
   1. Get a full understanding of the current situation. It helps to decide at the end of this phase whether the project should be moving to alpha
      
   2. If made correctly, this will limit the risk of failure and create a clear account of what should be done in alpha
      
2. Alpha
   1. It is the moment when the team can try different options
      
   2. At the end of the alpha phase, the team should have a prototype representing the future service
      
   3. Sometime after completing an alpha phase, the team may realize that it needs to revisit original objectives and needs
      
3. Beta
   1. Building stage of product development for the backend and the front end to provide a minimum viable product/service (MVP/MVS)
      
   2. At the end of the beta phase, the end-to-end minimum viable product should be ready to move to the private beta phase
      
4. Private Beta
   1. Way to put live the new service with a small number of users
      
5. Live
   

B. How research can fit into this framework?

Discovery: this phase is an excellent time to do competitive research

Alpha: In this phase, one can conduct pop-up research, also called guerrilla research which is a quick user testing session in which you test screens and prototypes with real users in their natural environment

Beta: One can test prototypes and solutions

Image 1: Integrating User research during PDLC. Credit Dr.Emmanuelle Savarit

The following diagram summaries research methods typically used in developing digital products

Image 2: Methods used in User Research. Credit Dr.Emmanuelle Savarit

How can one arrive at the Research maturity stage for the organisation since it’s so critical for continued success of the function? 

Author derives a version from the famed Nileson Norman corporate maturity model for UX

Image 3: Nileson Norman corporate maturity model for UX. Credit Dr.Emmanuelle Savarit

While rest of the stages are self-explanatory, in Skunkworks user experience the organization realizes that relying on design team intuition is risky, and the organization starts requesting some data

Image 4: Savarit User Research Maturity Model. Credit Dr.Emmanuelle Savarit

The author opines about a new trend in user research org: research ops team! It needs to be put in place once the organization already has a certain degree of UX maturity and a good understanding of the differences and advantages of conducting research in the product development life cycle. Typically sometimes user research leadership dons this role depending on size of the company, number of user researchers and project complexity!

C. How to prepare for User Research?

1. Identify if you need user research for your product
   
   1. The first question to ask is “Does my product/service involve users?” f it does not include end users, you will probably not require any user research.

2. Review the research that was done in the past and identify the gaps
   
   1. Identifying the gaps is the starting point of conducting user research and finding out what you don’t know
      
3. Identify your users
   
4. Build a case to get budget for your project
   
   1. While building the case to stakeholders, one has to make sure that they are not reinventing the wheel. One needs to have a clear account of what they are trying to do and why, what the benefits are of doing it, and what the risks are of not doing it
      
5. Get your stakeholders on board to support the project
   
6. Put in place the relevant capability to conduct the research
   

D. Research Preparation

This is the most important part of user research

Image 5: Steps in a Research Plan Document. Credit Dr.Emmanuelle Savarit

Few points to remember here

1. The methods to collect data are different from the research method that one may be using. The approach to collecting your data is not the way you are going to analyze the data. Following diagram provides a good heuristic

Image 6: Methods to Collect Data, depending on the development phase of the Project. Credit Dr.Emmanuelle Savarit

2. Every participant has the right to withdraw at any time during the research. This should be respected
   
3. Use a screener / questionnaire to recruit people for research. However be wary of panels since sometimes although using a panel is cheaper, but it may provide only those people who spend a lot of time taking part in studies, user research, and marketing just for the incentive
   
4. It is easier to get participants if you are giving an incentive
   
5. Organizing all documents in a clear format may save the team a lot of time
   

E. Research Methods // Quantitative Research Methods

Point to note here is that quantitative approaches used in isolation will provide only validation of preconceived ideas

Analytics: few points to note

1. Business context is important while analysing data. For example, in grocery shopping, it takes more than two minutes to fill up a shopping basket. So time spent metric has to be interpreted carefully vis-a-vis other categories
   
2. It’s important to realise that analytics alone is not enough, and does not answer the questions of why, how, and who
   

Surveys: Generally used to describe a population. A survey is a methodology that uses questionnaires to collect data

1. The advantage of conducting a survey is that one can get a large sample size that will validate some of the assumptions
   
2. Surveys are useful to validate findings from qualitative research or for data validation
   
3. One can also pay for a panel; useful if looking for a specific audience that is difficult to reach
   

Card Sorting: It’s asking intended readers to sort items into groups or categories that make sense to them. Few online tools such as Optimal Workshop’s Optimal Sort or UserZoom’s Card Sorting can be used here. Types of card sorting can be of three types: open, closed, hybrid. Some of the use cases for using card sorting quantitative research techniques are

1. Designing a new information product or improving one already existing: Open card sorting
   
2. Whether different groups of readers use the same approach to finding items: Open
   
3. Where new topics fit into an existing website or intranet: Closed
   
4. Whether you can reduce the number of categories, by testing whether a smaller set still works well: Closed
   
5. Set the tone of the organizational pattern while encouraging participants to generate and name their own categories: Hybrid
   
6. Further explore groupings that were unclear in an earlier sort while providing categories for groupings that were clear: Hybrid
   

Having ~40 cards ensures that one will generate the data needed to make decisions about Information Architecture. Beyond 60 cards, participants will be less likely to complete the card sort

Also one must randomize the order in which participants see the cards and the categories unless there’s a specific order to test. This removes list-primacy bias from the overall result

Two of the techniques to interpret data from card sorting are: dendrogram and similarity matrix

F. Research Methods // Qualitative Research Methods

A qualitative approach requires a small sample however analyzing qualitative data takes more time than the data collection itself. Some of the methods used to collect qualitative data are contextual inquiries/ethnography, interviews, focus groups, user testing, and diary studies

• To have reliable results with qualitative research, it’s recommended to have a minimum of eight people if you are conducting user testing every two weeks
  
• When product is live or in private beta and if one wants to test the end-to-end journey before release, it’s recommended to have between 12 to 24 participants

Contextual inquiries/ethnography

Here the researcher conducts a face-to-face semi-structured/conversational interview within a user’s natural environment. It’s an observational approach. It’s recommended to be used at the start of every project

1. Understanding their day and which tasks they have to complete at different times helps us understand which information they will require at time t
   
2. It enables us to capture not only how they are using the current product but also to identify the other tools/products that they are using
   
3. It however can be time-consuming and is more challenging to record

Interviews

User interviews are frequently used to capture general information about users. There are three types of interviews: structured (pre-defined response), semistructured, and unstructured (natural conversation)

Focus Group

It’s a session that lasts between one and two hours and takes place with six to ten participants who discuss a specific topic. It could be useful when you want your stakeholders involved in the project

Usability Testing

Usability testing/user testing is a face-to-face session with real users who are interacting with the product not only to evaluate the functionalities and the efficiency but also to capture the user’s behavior. However Usability is associated with quantifying success rate, task on time or error rates or satisfaction. User testing can be done as soon as there are some screens available. This is also useful for launching new websites and for evaluating end-to-end user journeys

Some techniques are A/B testing, eye tracking (when there is lot of information – tool Tobii Pro), guerilla testing or pop up research and System Usability Scale (SUS)

System Usability Scale

It is a quick usability test that is now a UX standard. It provides a high-level satisfaction score of the usability of a site, application, or any technological item. SUS is a simple, ten-item scale giving a global view of subjective assessments of usability

SUS scores have a range of 0 to 100. It is not a percentage. A score above 68 is above average, and a score below 68 is below average

Image 7: Interpreting SUS: gold standard for usability testing. Created by John Brooke (1986). Credit Dr.Emmanuelle Savarit

Diary Study

This approach permits us to get insights from users while they are experiencing a situation. The study is longitudinal and captures temporal information

1. Diary studies are used a lot in the field of education to understand the learning process or seeing how streaming services are being watched
   
2. It helps us uncover if participants develop any habits over time
   
3. It also allows a peak into user impressions as well as their frustrations
   
4. This technique is becoming more democratic now with proliferation of smartphones however making sure that participants remain proactive is a challenge for user researchers
   

How do we analyse Qualitative Data?

Affinity Diagram

A fast way to analyze data is to create an affinity diagram. Jiro Kawakita created this method that simplifies a large amount of data by grouping it by the themes that emerge from the data

1. However this analysis should be done with as much objectivity as possible
   
2. This method is suitable for brainstorming or early-stage analysis to identify the different steps of a user journey

Thematic Analysis vs. Content Analysis

1. If we identify that the user makes the same comments while booking a flight or conveys the same issues or frustration, that means we have identified a systematic pattern
   

Difference between User Profile and User Persona

A user profile is a set of characteristics based on demographics, job role, age, the industry the user is working in. A persona is a typical user who has specific characteristics such are interests, goals, behavior, attitude, habits. A persona is not a real user, it is not a case study, and it is not an imaginative character

G. Few ethical questions in User Research

• How do we make a website or a digital product accessible so that people with a disability can use it in terms of perception, comprehension, navigation, and interaction?
  
• Making respondents comfortable, taking their consent for data use, signing up NDAs for protecting company’s information and also understanding that their participation in the research is voluntary, and one cannot force them to respond to our questions
  

To sum it up, a general process for conducting user research can be summed up as below

• Get a brief
  
• Refine the scope of the research in order to set up stakeholder expectations
  
• Work with your stakeholders to draw up the research questions
  
• Choose the relevant method to answer the research questions
  
• Identify the right participants
  
• Collect and analyze the data
  
• Extract meaningful and actionable findings
  
• Share your findings with your team and stakeholders
  

And this process mostly remains the same across industries, org maturity levels, engagement model of the user researcher or geographies!