Distributional ratings, judgment decomposition, and their impact on interrater agreement and rating accuracy.

Abstract

One strategy suggested for improving the accuracy of the complex evaluative judgments involved in performance evaluation is to decompose them into a series of simpler judgments. Another is to collect observations in a distributional rating scheme in which raters estimate the frequencies of different classes of behavior and performance is assessed in terms of the relative frequencies of effective and ineffective behaviors. In the present study, we compared distributional ratings to Likert-type ratings of videotaped lectures at 3 levels of dimensional decomposition; ratings were evaluated in terms of inter rater agreement and rating accuracy. Decomposition led to increased agreement and accuracy, but the use of distributional ratings did not. The practical implications of the results are discussed.This study was conducted in fulfillment of the requirements for Robert A. Jako's master of science degree from Colorado State University.We thank Douglas Reynolds for comments and suggestions on earlier drafts of this article.

From the Work One Knows the Worker: A Systematic Review of the Challenges, Solutions, and Steps to Creating Synthetic Validity

“…As Harvey et al (1994) reviewed, holistic measures of measuring job performance are not equivalent to the decomposed methodology recommended here. As they and others have argued (e.g., Jako & Murphy, 1990), limitations in human information processing ability make holistic rating tenuous and thus are not recommended. Consequently, decomposed performance ratings should actually lead to higher observed validity coefficients.…”

Calculate or wait: Is man an eager or a lazy intuitive statistician?

“…Research concerning knowledge of distribution shape has mainly investigated if people can estimate the distribution shape of variables they have encountered in their everyday lives (Fox & Thornton, 1993;Griffiths & Tenenbaum, 2006;Jako & Murphy, 1990;Linville, Fischer, & Salovey, 1989;Nisbett, Krantz, Jepson, & Kunda, 1983;Nisbett & Kunda, 1985). Nisbett et al (1983), for example, asked students to estimate the distribution of grade point averages among their peers, and Griffiths and Tenenbaum (2006) were concerned partly with distributions of baking times of pastries and movie runtimes.…”

Problems With the Use of Profile Similarity Indices in the Study of Congruence in Organizational Research

“…However, many of these problems also arise when PSIs are used for other purposes. For example, the assessment of interrater agreement has often relied on PSIs, including Q (e.g., Harvey & Hayes, 1986;Jones et al, 1983;Sanchez & Levine, 1989;Bui & Barry, 1986), JDI (e.g., Jako & Murphy, 1990;Zalesny & Kirsch, 1989), and various indices not reviewed here, such as the intraclass correlation coefficient (e.g., Jako & Murphy, 1990;Jones et al, 1983;Sanchez & Fraser, 1992) and rwG (e.g., James, Demaree, & Wolf, 1984). These indices collapse across multiple raters, multiple rating dimensions, or both, thereby concealing whether agreement is uniform across raters and dimensions or limited to some subset of raters or dimensions.…”