The Relationship Between Levels of User Experience with a Product and Perceived System Usability

Abstract

The System Usability Scale (SUS) is a ten-point assessment tool developed as a reliable low-cost subjective usability scale that can be applied to systems in any number of contexts. Research has demonstrated higher usability ratings from users who claim greater experience with an interface than from those who rate themselves as having less experience. This paper describes research to extend this work by experimentally controlling the experience levels of the users over the course of the study, rather than relying on users' self-report. Two studies were conducted. In the first, Microsoft Publisher was used over three one hour sessions, with usability being measured with the SUS at the completion of each session. In the second study, MathWorks MATLAB was used over the course of 14 weeks, and SUS usability was measured near the beginning, the middle and end of this time frame. Results from the MS publisher study showed an increase in reported usability with increased experience consistent with the literature, but the data from the MATLAB study did not show this trend. Reasons for this discrepancy are discussed, as are future research directions that could shed further light on these unexpected findings.

Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience

“…In tune with previous studies (Kortum & Johnson, 2013;McLellan et al, 2012;Sauro, 2011), users with a greater amount of product experience were more satisfied than users with less experience. This was also confirmed by the ANOVAs performed for Studies 1 and 2: Greater product experience was associated with a higher the level of user satisfaction.…”

“…Therefore, consistent with previous research (Kortum & Johnson, 2013;McLellan et al, 2012;Sauro, 2011), to obtain an exhaustive picture of user satisfaction, researchers and practitioners should take into consideration each user's amount of exposure to the product under evaluation (duration and frequency of use).…”

“…Currently, three studies (Kortum & Johnson, 2013;McLellan et al, 2012;Sauro, 2011) have investigated the relationship between SUS scores and amount of product experience. The results of these studies have consistently indicated that more experienced users had higher satisfaction outcomes (SUS scores).…”

“…In summary, researchers and practitioners assess user satisfaction by means of questionnaires, but there are only a few empirical studies that have systematically analyzed the variation of the outcomes of satisfaction scales when filled out by users with different amounts of experience in the use of a product (Kortum & Johnson, 2013;Lindgaard & Dudek, 2003;McLellan et al, 2012;Sauro, 2011).…”

The Relationship Between System Effectiveness and Subjective Usability Scores Using the System Usability Scale

“…Because usability professionals typically do usability testing in the field (vs. a lab), it is important that any research done on usability tools be validated in the field. Further, previous research into the SUS (Kortum & Johnson, 2013) has noted that there can be significant differences in subjective usability scores when they are measured in the field rather than the laboratory. To make sure that a full understanding of the true relationship between SUS scores and success rates was obtained, we performed two different studies to measure this relationship.…”