Some factors influencing the unreliability of clinical judgments

Abstract

PROBLEMThe unreliability of clinical judgment is well known, particularly in the field of diagnosis(l1 8). Little has been done, however, toward any analysis of the factors responsible for this unreliability in order that it may be understood, controlled and corrected. Magaret ('I) points out that we need both a philosophy of diagnosis and a sophisticated understanding of its nature. Hunt, Wittson and Hunt (4* 6 , suggest that our understanding of clinical judgment might be furthered if we conceived of it, not as a unique and special kind of professional performance, but as one example of the broader phenomenon of human judgment in general. The present study takes this approach, and studies the effect upon the clinical judgment of both the professional experience of the judge, and of anchoring the scale which he is using to make his judgment. Both lo) and anchoring c3, 9, have been shown to influence judgment in a wide range of situations varying from those of classical psychophysics to the judgments of the prestige of occupations and the undesirability of certain forms of behavior.We assumed that clinical judgments might show the same relativism that has been demonstrated in other fields of judgmeqt and that this relativism might be contributing to the unreliability of the clinical judgments. Specifically we proposed bhree hypotheses :(1) That introducing an anchoring stimulus at either end of the stimulus continuum would cause a shgt in the judged value of the stimuli being evaluated;(2) That these anchoring effects would be a function of the experience of the judges, with the most experienced judges showing the least shift; and(3) That the reliability of the judgments, here defined as inter-judge agreement, would also be a function of experience, with the most experienced judges showing the greatest reliability or agreement. SUBJECTS AND MATERIALSI n order to test the effect of experience, three groups of judges were selected from three separate levels of clinical training. Sixty were undergraduates who had just completed a course in abnormal psychology; sixty were graduate students interning during a clinical psychology training program, and sixty were professional clinicians with four years or more on-the-job professional experience. As stimuIi, schizophrenic responses to items on the Wechsler-Bellevue and Terman-Binet vocabulary tests were used and the subjects were asked to rate these on an 11-point scale for the severity of the disorder in the thinking processes exhibited in the responses. PROCEDUREFor the construction and equation of the two stimulus series used in the experiment, it was first necessary to obtain a number of stimuli whose stimulus values *This paper is a condensation of a longer one submitted to the Graduate School of Northwestern University in partial fulfillment of the requirements for the Ph.D. degree. The complete thesis, containing a more detailed statement of procedure and all the relevant data with their complete st,atistical analyses, is available from University Microfilms, Ann...

Computer-Based Role-Playing for Interpersonal Skills Training

“…These results differ from those found in previous research, which showed that peer-ratings were as good in discriminating the level of social skill as trained judges' ratings of specific behavioral indices (Firth, Conger, Kuhlenschmidt, & Dorcey, 1986). In most studies reported, naive raters have served as raters because it was assumed that they approach to the scaled dimensions as taught, rather than to their intuition (Arnhoff, 1954;Cronbach, 1960). For the experiment, reported in Chapter 7, two changes are made.…”

Psychodiagnostic preference by professional affiliation and length of experience

“…Arnhoff ( 2 ) found that inter-judge agreement on judging amount and kind of psychopathology decreased with amount of professional experience, although a repetition of Arnhoff's study(12), using revised instructions, reversed the results. Arnhoff ( 2 ) found that inter-judge agreement on judging amount and kind of psychopathology decreased with amount of professional experience, although a repetition of Arnhoff's study(12), using revised instructions, reversed the results.…”

Reducing the dependence of clinical judgment on the immediate context: Effects of number of categories and type of anchors.

“…Although these techniques often lead to higher reliability (Bernardin & Beatty, 1984), there is no indication that they reduce the effects of differences of the immediate context for judgment. In previous research on clinical judgment, use of detailed descriptions of each category has met with little success in reducing contextual effects (Arnoff, 1954;Perrett, 1971); however, the results of Experiment 1 suggested circumstances under which anchoring might prove effective. Experiment 2 investigated these conditions using two different techniques for anchoring the scale of judgment.…”

“…In experiments using clinical judgment, descriptive anchors have not succeeded in reducing contextual effects (Arnoff, 1954;Perrett, 1971). However, comparison between the results of Experiment 1 and Perrett's (1971) study suggested that providing scale anchors might reduce the effects of placing the target stimuli last for restricted-range conditions.…”