Enhancing Accuracy in Observational Test Scoring: The Comprehensive System As a Case Example

Abstract

Inaccuracies in administration and scoring can potentially compromise the validity of any standardized psychosocial measure. The threat is particularly pertinent to methods involving behavioral observation, a category that includes many intelligence tests, neuropsychological measures, personality assessment instruments, and diagnostic procedures. Despite evidence and conjecture that errors in testing procedure are common for at least some of these measures and that these errors are often severe enough to influence interpretation, the topic has received relatively little attention. In particular, the absence of any safeguard against inaccurate test use in clinical situations can put the respondent at risk and violates ethical standards for the use of tests. In this article, I review some issues surrounding accuracy in testing procedures, including a discussion of what is known about the problem, an evaluation of several approaches to improving testing practices, and a review of recommendations for the statistical evaluation of rater accuracy. In this article, I use the Rorschach Comprehensive System (Exner, 1993) to demonstrate the concepts discussed.

Rorschach Comprehensive Variables and Assessment for Psychodynamic Psychotherapy

“…This should also lessen the risk that statements actually not relevant for psychotherapy planning influence decisions with unfortunate consequences for the patient. However, this also requires correct administration and scoring of the Rorschach, not only in research but also in clinical practice; McGrath (2003) stressed that it is essential to avoid idiosyncrasies in scoring and administration. A psychodynamic psychotherapy is a major and important personal undertaking mainly for the patient but also for the therapist, and the time spent in careful assessment and planning to avoid unintended consequences is worth the costs of time and other resources.…”

The Forensic Psychologist as Consultant: Examples from a Jurisprudent Science Perspective

“…Scoring errors are a significant issue with cognitive testing (Alfonso, Johnson, Patinella & Rader, 1998;Belk, LoBello, Ray & Zachar, 2002;Hall, 1999;Hopwood & Richard, 2005;LoBello & Holley, 1999;Sullivan, 2000) and personality testing alike (Allard & Faust, 2000;Goddard, Simons, Patton & Sullivan, 2004;Guamaccia, Dill, Sabatino & Southwick, 2001;McGrath, 2003). Such problems occur with computerized scanning as well as with hand-scoring (Marcus & Robins, 1998), and have been documented in countries around the world (Goddard, Simons, Patton & Sullivan, 2004;Simons, Goddard & Patton, 2002;Sullivan, 2000 Of course, scoring errors are not confined to bored or careless licensees at the doctoral level.…”

“…Researchers have proposed numerous training strategies to enhance scoring accuracy (Egan, McCabe, Semenchuk & Butler 2003;Hall, 1999;McGrath, 2003), and computer scoring may help to "eliminate reliability problems associated with scoring complexity" (Allard, Butler, Faust & Shea, 1995, p. 308). University training programs and the agencies that employ students-sometimes for a stipend, and sometimes not-might well enquire: "How else are students to be trained?"…”

Rorschach Assessment in Child Custody Case

“…Of further note in this regard, moreover, research findings indicate that coding errors in clinical settings can be reduced by conscientious recourse to detailed coding criteria and guidelines that are readily available in textbooks and workbooks (McGrath, 2003). Accordingly, instances in which assessors in the field code Rorschach responses inaccurately, and thereby fail to reach substantial agreement with each other, would seem more likely to reflect insufficient preparation and care on part of examiners than any inherent shortcoming of the instrument.…”