Abstract
User trials provide valuable information on how users respond to interfaces in practice. However, it can be hard to ensure a representative sample. We propose a methodology to improve the understanding of the sample’s skew and to identify the characteristics of those who are missing. This can improve the interpretation of results and inform further recruitment to improve the sample. The methodology involves comparing samples with survey data from the UK population on technology experience, competence and attitudes. We provide a case study of this methodology in practice. 30 participants were recruited using quota sampling with significant effort to obtain people with low technology experience. Nevertheless, comparison with the survey data identified four key groups of people not included in the sample, covering 29% of the population. We discuss how these missing people would likely respond on the tasks, based on the characteristics of similar people in the survey.
Keywords
user trials, survey data, sampling, inclusive design
DOI
https://doi.org/10.21606/drs.2022.676
Citation
Petyaeva, A., Goodman-Deane, J., Bradley, M., Waller, S., and Clarkson, J. (2022) Improving our understanding of user trial samples using survey data, in Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.), DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. https://doi.org/10.21606/drs.2022.676
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Conference Track
Research Paper
Included in
Improving our understanding of user trial samples using survey data
User trials provide valuable information on how users respond to interfaces in practice. However, it can be hard to ensure a representative sample. We propose a methodology to improve the understanding of the sample’s skew and to identify the characteristics of those who are missing. This can improve the interpretation of results and inform further recruitment to improve the sample. The methodology involves comparing samples with survey data from the UK population on technology experience, competence and attitudes. We provide a case study of this methodology in practice. 30 participants were recruited using quota sampling with significant effort to obtain people with low technology experience. Nevertheless, comparison with the survey data identified four key groups of people not included in the sample, covering 29% of the population. We discuss how these missing people would likely respond on the tasks, based on the characteristics of similar people in the survey.