An Examination of Interrater Reliability for Scoring the Rorschach Comprehensive System in Eight Data Sets

Abstract

In this article, we describe interrater reliability for the Comprehensive System (CS; Exner. 1993) in 8 relatively large samples, including (a) students, (b) experienced re- searchers, (c) clinicians, (d) clinicians and then researchers, (e) a composite clinical sample (i.e., a to d), and 3 samples in which randomly generated erroneous scores were substituted for (f) 10%, (g) 20%, or (h) 30% of the original responses. Across samples, 133 to 143 statistically stable CS scores had excellent reliability, with median intraclass correlations of.85, .96, .97, .95, .93, .95, .89, and .82, respectively. We also demonstrate reliability findings from this study closely match the results derived from a synthesis of prior research, CS summary scores are more reliable than scores assigned to individual responses, small samples are more likely to generate unstable and lower reliability estimates, and Meyer's (1997a) procedures for estimating response segment reliability were accurate. The CS can be scored reliably, but because scoring is the result of coder skills clinicians must conscientiously monitor their accuracy.

Introducing a Bayesian Approach to Determining Degree of Fit With Existing Rorschach Norms

“…Lastly, some of our inter-rater reliability values were relatively low, compared to other data reported in the literature (e.g., Meyer et al, 2002;Viglione, Blume-Marcovici, Miller, Giromini, & Meyer, 2012;Viglione & Taylor, 2003). Though both the original and re-scored records were coded by (graduate) students, the initial coding was also carefully supervised by a senior clinician and researcher who has been using the Rorschach for years.…”

An Inter-Rater Reliability Study for the Rorschach Performance Assessment System

“…A great deal of research with various Rorschach systems and scores has demonstrated strong inter-rater reliability (Acklin, McDowell, Verschell, & Chan, 2000;McDowell & Acklin, 1996;Exner, 1993;Meyer & Archer, 2001;Viglione, 1999;Viglione & Meyer, 2008;Viglione & Taylor, 2003). Inter-rater reliability for the CS (Exner, 2003) has been particularly wellestablished, with a large-scale meta-analysis of CS inter-rater reliability reporting high inter-rater reliability for the majority of CS scores (specifically, median intraclass correlations for statistically stable scores ranged from .72-.96; Meyer et al, 2002). 1 This range of reliabilities has been considered "good to excellent" by some authors (Cicchetti, 1994;Shrout & Fliess, 1979) and "acceptable to excellent" by others (Hunsley & Mash, 2008).…”

Assessing Implicit Motives in U.S. College Students: Effects of Picture Type and Position, Gender and Ethnicity, and Cross-Cultural Comparisons

“…Interrater reliability, assessed by the intraclass correlation coefficient (ICC), of coder A with coder C was greater than .70 for all three motives and can thus be considered good to excellent (cf. Meyer et al, 2002; Table 1 reports detailed ICC summary information). We also determined the participants' levels of activity inhibition in the U.S. sample by counting the frequency of the word not (cf.…”