A characteristic of User Experience Design that differentiates it from other forms of design is its data gathering and user testing processes. Where other forms of design tend to be based around a more direct client brief-to-three rounds of revisions model; User Experience designers employ a more inquisitive approach that attempts to dispel preconceived assumptions a designer might have about how something works and instead uncover insights into the core of a problem by listening and understanding the people who will actually interface with the final design (a.k.a the end-user). This more Human-Centred-Design (HCD) approach results in designs that reflect their user’s demographics and needs, creating designs which are more suitable and appropriate than more traditional top-down design approaches.
Defining Goals & Identifying Participants
When starting the data gathering process, it is important to first narrow your scope by clearly defining your goals for the research process. Put simply, we need to first answer the following questions:
- What design phase are we? Different stages of the design process will require different insights from users.
- What do we want to learn at this phase? During earlier phases in the design process we’ll want to learn as much as we can about the end-user, while in later phases it’s important to validate design decisions through user-testing or observation.
Defining user research goals will help in choose what data gathering methods to use and what kind of information (quantitative or qualitative) would be most useful to derive from the process.
Since the “main reason for gathering data is to glean information about users, their behaviours, or their reaction to technology… it is important to set specific goals for the study. These goals will influence the nature of the data gathering sessions, the data gathering techniques to be used, and the analysis to be performed.” (Chapter 8, Interaction Design: Beyond Human-Computer Interaction)
It is similarly important to also identifying suitable participants for the data to be gathered. The data gathering goals will dictate the types of people from whom the data should be gathered. The individuals who fit this profile are called the ‘population’ or ‘study population’. How a study population is defined can have significant implications for what the gathered data tells us; gathering data from the wrong population or by incorrectly ‘sampling’ the correct population will skew our research findings.
For this reason, where participants are to be chosen for a study, it is important that they adequately represent the study population. This is called sampling and there are three different kinds:
- Saturation Sampling refers to the rare situation where all members of a population are part of the sample that is studied (i.e. when designing tailored accounting software for a small business, it is likely all employees who would use it would be interviewed).
- Probability/Stratified Sampling refers to random sampling either every nth member of a population or by dividing a population into groups and then randomly sampling within those groups.
- Non-probability Sampling refers to connivance sampling or volunteer panels where participants self-nominating to participate in the study.
“The crucial difference between probability and non-probability sampling is that in the former you can apply statical test and generalise to the whole population, while in the latter such generalisations are not robust.”
In terms of how big your sample should be, it depends on what techniques you’re using and what you’re studying. For most cases, user experience and interaction designers will usually take input from at least 5 and a median of 12 participants. (Chapter 8, Interaction Design: Beyond Human-Computer Interaction)
Quantitative Data (= Numbers) and Basic Analysis
Quantitative data is any data that is in a numerical form or data that can be directly translated into numbers (such as statistics and percentages). The objective of quantitative research is to make empirical observations that can then be expressed mathematically to ascertain the magnitude, amount, or size of something as well as uncovering quantitative relationships between datasets.
When analysis quantitative data, it is important to consider whether or not a particular analysis is meaningful in the specific context. Similarly, it is important to consider whether or not expressing a particular dataset in particular way is appropriate for achieving the research goals.
Basic Quantitative Data Analysis
There are many ways to statistically analyse quantitative datasets; including percentages, averages, and more advanced statistically trend analysis using specialised software packages such as R. For basic user experience research purposes, it is usually enough to identify relevant groupings using percentages or basic trends (i.e. 1 in 5 users), as well as identifying averages.
There are three different types of averages which are useful for statistical analysis:
Mean – The mean is what is typically understood as being an ‘average’, that is the value derived from adding all figures together and then dividing by the amount of figures added (i.e. Mean = [1+2+3+4+5]/5 = 3). It represents the central value of all the figures.
Median – The median is the middle figure when all the data when all figures are ranked from smallest to highest. It represents the “middle value” between the lower and higher half of a dataset (i.e. 1,3,7,9,10 = Median = 7). The benefit of using a Median to describe data is that “it is not skewed by a small proportion of extremely large or small values, and therefore provides a better representation of a ‘typical’ value” (Wikipedia).
Mode – The mode is the most occurring figure in a dataset, it is the number that appears most often in a dataset and represents the value that is most likely to be sampled (i.e 1,3,3,7,9 = Mode = 3).
Qualitative Data (= Observations) and Basic Analysis
Qualitative Data is data obtained from first-hand observation, interviews, questionnaires (on which participants write descriptively), focus groups, participant-observation, recordings made in natural settings, documents, and artefacts. It is a means by which to gain a more complete understanding of an individuals’ social reality in a way quantitative research simply cannot (because it’s hard to numerically quantify the broad spectrum of human emotion and experience). For this reason, qualitative studies attempt to gain an understanding of something by focusing on its nature and can be represented by themes, patterns, and stories.
Qualitative data can be analysed either inductively, where ideas are extracted from the data, or deductively, where data is organised according to preconceived theoretical or conceptual ideas. When beginning to analyse a qualitative dataset, it is a good idea to first gain an overall impression of the data and observe things that stand out such as interesting features, topics, and repeated observations.
Basic Qualitative Data Analysis
There are a number of different ways to analyse qualitative datasets, often times these methods are used together:
- Identifying Themes/Thematic Analysis & Affinity Diagrams – Thematic analysis aims to identify, analyse, and report patterns in a dataset. A ‘theme’ can be identified as anything important about the data (i.e. a topic of discussion, or a feature of the collected data) as it relates to the goals of the research. Themes may relate to a wide range of aspects, including: behaviour, a user group, events, places or situations, etc. Once a number of themes have been identified and categorised, an attempt is made to observe an overarching narrative to the data. Affinity Diagrams are often used to aid the process by grouping notes into similar categories which aren’t predefined but rather emerge from the data.
- Categorising Data – Another analytical technique involves parsing a dataset (such as an interview transcript) according to predetermined categories (such as ‘user interface problems’). The themes observed in the data are bullet pointed under the predefined categories, providing a comprehensive list of all the ideas that arose during the research session. This technique is less suitable for exploratory research as themes that can emerge from the research are fixed but offers a meaningful way to structure data where specific insights are needed (such as during user testing).
- Critical Incident Analysis – This final analytical technique provides a means through which to reduce the information being processed from a dataset by focusing only on the points that a relevant to the research goals. Put simply, researchers parse a dataset by focusing on observing only behaviour that relates to the goals of the activity instead of observing interpretations, ratings, and opinions.
Data Gathering Methods
- Questionnaires & Surveys – Surveys are an effective way of gathering data that can be represented in a quantitative way (i.e. percentages or statistics) using close-ended questions, although they can also be used to gather qualitative data (using open-ended questions). Surveys and questionnaires are usually used to “build or validate user models like segmentation or personas” (A Project Guide to UX Design) and to measure user satisfaction with an existing or new product. It is a powerful method for supplementing and validating quantitative data gathered during user-interviews and contextual inquiry.
- User Interviews – Interviews are structured conversations with people in the sample group and are a valuable way of gaining an understanding of user’s preferences and attitudes (which is a rich dataset that can be used to create personas). When conducting an interview, it is best to focus on the participants existing personal experience with open-ended questions and not ask leading questions or those that ask the participant to speculate on future behaviour. Interviews are a great tool for learning more about users and their feelings.
- Contextual Inquiry– Contextual Inquiry combines user observations with either active or passive interview techniques to gain insight into the context of the user experience. Data is gathered in the environment most relevant to the subject being studied and interviewers can either actively ask questions during their observation or wait until after the session to question particular behaviours. It differs from Usability Testing by as users aren’t instructed on what tasks to perform but rather observed as they go about their work.
- Usability-Testing – Usability or User Testing involves observing users as they are asked to perform specific tasks on a product (either existing or a prototype) to “uncover usability issue and gather ideas to address them” (A Project Guide to UX Design). Participants are often observed while being asked to ‘think-aloud’ as to make their thought process (and issues they come across) obvious to the observer.
- Card Sorting – Card Sorting is an activity in which participants (either individuals or small groups) organise information into groups (either supplied or determined by the participant) using cards with relevant information on them. It is a useful method for structuring large data sets around user understandings. An example is using card sorting to structure a website’s site map.
- Focus Groups
References
The above summary represents notes from the following book chapters:
- Chapter 8, Interaction Design: Beyond Human-Computer Interaction
- Chapter 9, Interaction Design: Beyond Human-Computer Interaction
- Chapter 6, A Project Guide to UX Design