Measurement Practices in UX Research: A Systematic Quantitative Literature Review

Abstract

User experience research relies heavily on survey scales as an essential method for measuring users' subjective experiences with technology. However, repeatedly raised concerns regarding the improper use of survey scales in UX research and adjacent fields call for a systematic review of current measurement practice. Until now, no such systematic investigation on survey scale use in UX research exists. We, therefore, conducted a systematic literature review, screening 707 papers from CHI 2019 to 2021, of which 207 were eligible empirical studies using survey scales. Results show that papers frequently lacked rationales for scale selection (70.05%) and rarely provided all scale items used (21.74%). Nearly half of all scales were adapted (44.69%), while only one-fifth of papers reported any sort of scale quality investigation (19.32%). Furthermore, we identified 224 different scales and 287 distinct constructs measured. Most scales were used once (76.79%), and most constructs were measured once (81.88%). Results highlight questionable measurement practices in UX research and suggest opportunities to improve scale use for UX-related constructs. We provide recommendations to promote improved rigor in following best practices for scale-based UX research. This research was funded internally. All materials are available on OSF: https://osf.io/dgq6b/

Development and Validation of a Positively Worded German Version of the System Usability Scale: The SUS-DE-Pos

“…Thus, five positive alternatives, one for each reverse-coded SUS item, were chosen based on the results from the first study and the language evaluation. Finally, a second study using of survey scales (e.g., Perrig, von Felten, et al, 2023;Salazar, 2015), they can also lead participants to misrespond if their wording leaves room for interpretation, resulting in comprehension issues among respondents (Sauro & Lewis, 2011;Weijters & Baumgartner, 2012). Evidence of such misresponses was also observed in the results of them to avoid negatively formulated items and, thus, the issues associated with their use.…”

“…Thus, a second online study with a considerably larger sample using a set of existing websites as stimuli was conducted. The procedure and design of this second study closely followed prior research on the development of a positive version of a visual aesthetics scale (Perrig, von Felten, et al, 2023). Alike to the first study, this study was preregistered on OSF (https://doi.org/10.17605/OSF.IO/Z7YBR).…”

“…Following the procedure from Perrig, von Felten, et al (2023), we prepared a set of 12 existing Swiss websites with content in German as stimuli. We chose to use Swiss websites to minimize the risk that participants were already familiar with the sites while still allowing the German participants to be able to understand the website's language.…”

“…They further demonstrated that this positive-only scale version yields results comparable to the original version. A similar project for a website aesthetics scale has shown that switching from negative to positive item wording can improve scale quality (Perrig, von Felten, et al, 2023). Past work has highlighted several issues with negatively worded items, including lower scale reliability (Barnette, 2000), distorted factor structure (Schmitt & Stuits, 1985), and reduced cross-cultural applicability (Wong et al, 2003).…”