The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Abstract

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking-aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.

Identifying usability and fun problems in a computer game during first use and after some practice

“…While it is possible to perform a test to determine quantitative measures like efficiency, effectiveness, and satisfaction (ISO, 1998), another common goal is to identify specific parts of a system that cause users trouble (Hertzum and Jacobsen, 2001) in order to improve the system. This is often called formative testing (Barnum, 2002).…”

“…However, research has shown that the set of identified problems depends both on the users taking part in the user test (Nielsen, 1994), and the evaluators analysing the data (Jacobsen, 1999;Hertzum and Jacobsen, 2001). Usually, products are tested with users who use these products for the first time, but it could be expected that there are differences in users' behaviour when they have become more familiar with a product.…”

Correlation between safety assessments in the driver–car interaction design process

“…This may be explained as an evaluator effect, since the Nielsen heuristics rating involves interpretation of the heuristics (Hertzum and Jacobsen, 2003). It is interesting to note that customer ratings, time on task measurements and mental workload measurements yield the same results, while the Nielsen heuristics rating does not.…”

“…It is interesting to note that customer ratings, time on task measurements and mental workload measurements yield the same results, while the Nielsen heuristics rating does not. The latter has also been questioned in terms of the type of problems that it detects (Wixon, 2003) and of evaluator effects (Hertzum and Jacobsen, 2003;Jeffries and Desurvire, 1992;Ling and Salvendy, 2009).…”

“…Owing to the functional revolution in modern cars, UEMs employed within the car domain now have similar requirements as those employed within the general HCI domain. In the HCI domain, much effort has been spent on securing the reliability and validity of UEMs (Gray and Salzman, 1998;Hartson et al, 2003;Hertzum and Jacobsen, 2003;Jeffries and Desurvire, 1992;Ling and Salvendy, 2009;Nielsen, 1994;Wixon, 2003). However, in the case of UEMs for IVIS, where the assessed systems are operated as secondary tasks and are thus central to safety, more effort has been put into identifying safety related problems in expert inspections (AAM, 2002;ESOP, 2006;Stevens et al, 1999) rather than verifying whether the problems encountered are found in user assessments.…”