Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience

Abstract

Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human-computer interaction literature the relationship between the outcomes of standardized satisfaction scales and the amount of product usage has been only marginally explored. The few studies that have investigated this relationship have typically shown that users who have interacted more with a product have higher satisfaction. The purpose of this article was to systematically analyze the variation in outcomes of three standardized user satisfaction scales (SUS, UMUX, UMUX-LITE) when completed by users who had spent different amounts of time with a website. In two studies, the amount of interaction was manipulated to assess its effect on user satisfaction. Measurements of the three scales were strongly correlated and their outcomes were significantly affected by the amount of interaction time. Notably, the SUS acted as a unidimensional scale when administered to people who had less product experience but was bidimensional when administered to users with more experience. Previous findings of similar magnitudes for the SUS and UMUX-LITE (after adjustment) were replicated but did not show the previously reported similarities of magnitude for the SUS and the UMUX. Results strongly encourage further research to analyze the relationships of the three scales with levels of product exposure. Recommendations for practitioners and researchers in the use of the questionnaires are also provided.

Survey of research on Quality of Experience modelling for web browsing

“…While satisfaction is considered to be a key element of the user experience as discussed by Borsci et al [12], Hassenzahl [41], and Lewis [68], it may also be incorporated in the meaning of QoE considering the notion of ''delight'', together with the notions of pleasure, functionality, excitement, etc. which constitute the user experience.…”

Biofeedback in partial weight bearing: usability of two different devices from patient's and physical therapist's perspective

“…1-100). Sauro and Lewis reported the overall mean score to be 68 [39,41]. In the current study, we considered SUS scores of at least 62.7 (corresponding with at least a grade C-) as acceptable usability.…”

“…The overall SUS score ranges from 0 to 100, where higher scores represent better usability. Based on SUS data from a large number of studies Sauro and Lewis produced norms for the interpretation of mean SUS scores, the Curved Grading Scale (CGS) [39,41]. The CGS assigns grades as function of SUS scores and ranges from F (absolutely unsatisfactory) to A+ (absolutely satisfactory), the grades are as follow: Grade F (0-51.…”

“…In the current study, we considered SUS scores of at least 62.7 (corresponding with at least a grade C-) as acceptable usability. The SUS is a simple and reliable method, widely used in usability evaluation [22,39,41,42]. The English SUS shows good internal consistency: between studies Cronbach's alpha ranged from 0.85 to 0.91 [42}.…”

Evaluation of the Impact of Usability in Arabic University Websites: Comparison between Saudi Arabia and the UK

“…The advantage of SUS is that it can be summarised as being a short list of questions and free to use [39]. The SUS questionnaire contains ten questions, after the participant is familiar with the website (spends twenty minutes before answering certain questions) they are asked to read each question carefully and fill the questionnaire by using the scale (five-point Likert-type scale) from "1" which is strongly disagree to "5" which is strongly agree.…”