Measurement practices in user experience (UX) research: a systematic quantitative literature review

Abstract

User experience (UX) research relies heavily on survey scales to measure users' subjective experiences with technology. However, repeatedly raised concerns regarding the improper use of survey scales in UX research and adjacent fields call for a systematic review of current measurement practices. Therefore, we conducted a systematic literature review, screening 153 papers from four years of the ACM Conference on Human Factors in Computing Systems proceedings (ACM CHI 2019 to 2022), of which 60 were eligible empirical studies using survey scales to study users' experiences. We identified 85 different scales and 172 distinct constructs measured. Most scales were used once (70.59%), and most constructs were measured only once (66.28%). The System Usability Scale was the most popular scale, followed by the User Experience Questionnaire, and the NASA Task Load Index. Regarding constructs, usability was the most frequently measured, followed by attractiveness, effort, and presence. Furthermore, results show that papers rarely contained complete rationales for scale selection (20.00%) and seldom provided all scale items used (30.00%). More than a third of all scales were adapted (34.19%), while only one-third of papers reported any scale quality investigation (36.67%). On the basis of our results, we highlight questionable measurement practices in UX research and suggest opportunities to improve scale use for UX-related constructs. Additionally, we provide six recommended steps to promote enhanced rigor in following best practices for scale-based UX research.

Development and Validation of a Positively Worded German Version of the System Usability Scale: The SUS-DE-Pos

“…Among the survey scales used in Human-Computer Interaction (HCI) and user experience (UX) research, the System Usability Scale (SUS) by Brooke (1996) is among the most popular (Bargas-Avila & Hornbaek, 2011;Perrig et al, 2024;Pettersson et al, 2018). Initially described as a "quick and dirty usability scale" (Brooke, 1996), the ten-item SUS has proven to be a quick but not "dirty" way to assess the perceived usability of users after interacting with a system (Lewis, 2018).…”

“…We make this recommendation mainly for the sake of consistency. If too many different German versions of the SUS are being used across the research field, the comparability of findings across research projects is reduced(Perrig et al, 2024). Thus, despite the slightly better CFA results for the SUS versions based onGao et al (2020), we recommend working with the SUS-DE-Rummel given that its wording is closer to the best-performing scale in the second study, the SUS-DE-Pos, with the exception of the negative item formulation.…”