Empirical Assessment of Expertise

Abstract

The assessment of expertise is vital both in practical situations that call for expert judgment and in theoretical research on the psychology of experts. It can be difficult, however, to determine whether a judge is in fact performing expertly. Our goal was to develop an empirical measure of expert judgment. We argue that two necessary characteristics of expertise are discrimination of the various stimuli in the domain and consistent treatment of similar stimuli. We combine measures of these characteristics to form a ratio we call the Cochran-Weiss-Shanteau (CWS) index of expertise. The proposed index was demonstrated using two studies that distinguished experts from nonexperts based on their judgmental performance. The index provides new insights into expertise and offers a partial definition of expertise that may be useful in a variety of theoretical and applied settings. Potential applications of this research include selection, training, and evaluation of experts and of expert-machine systems.

Assessing Professional Competence by Using Occupational Judgment Tests Derived From Job Analysis Questionnaires

“…According to this view, high levels of motivation may enhance the development of tacit knowledge by supporting reflections upon job-related experiences. Sound judgment also has been identified as the critical basis of much expertise in the human factor literature (Weiss & Shanteau, 2003); therefore, these theoretical frameworks support the general inference that judgment tests provide another method to assess occupational knowledge and competence. However, conventional SJTs have been expensive to develop and time-consuming to administer (McDaniel & Nguyen, 2001).…”

“…While this type of knowledge may not be necessary to enable job incumbents to complete well-defined tasks, broad understandings concerning the interdependencies of task and employee characteristics have been a hallmark of expertise and provide a solid foundation to excel in many domains (Chi, Glaser & Farr, 1988;Weiss & Shanteau, 2003). When conducting job analyses, it has been customary to survey job incumbents and technical experts for nearly a century (Brannick & Levine, 2002).…”

Examining Expertise Using Interviews and Verbal Protocols

“…The experience of the professional and the presence of professional criteria, such as degrees, licenses, memberships of professional organisations, prizes, and teaching experience usually work well to identify experts (Evetts, Mieg, & Felt, 2006;Hoffman, Shadbolt, Burton, & Klein, 1995;Mieg, 2006). There is a notable absence of a 'gold standard' of professional performance that is based on a validated objective outcome measure (Ericsson, 2004;Weiss & Shanteau, 2003; Shanteau, Weiss, Thomas, & Pounds, 2001) and one best solution or approach to a problem may not even exist (Tracey, Wambold, Lichtenberg, & Goodyear, 2014). In medicine, for example, physicians have to make decisions…”

The Role of Expertise Research and Human Factors in Capturing, Explaining, and Producing Superior Performance

“…Detailed observations of skilled performers engaged in representative tasks may lead to further refinements of training technologies and hardware/software interfaces that better reflect the abilities brought to bear in complex task environments. In turn, new measurement technologies developed by both hardware and software engineers may permit the development of reliable performance metrics in domains that presently lack objective measures of skill (e.g., Weiss & Shanteau, 2003). Despite the difficulty in pursuing research with experts, the studies undertaken in domains such as sports, medicine, and aviation during the past decade show the promise of the expert performance approach.…”