Methodological considerations in validating computer-based test interpretations: Controlling for response bias.

Abstract

Although computer-based test interpretation (CBTI) systems have been operational for nearly 25 years, their availability and adoption in routine clinical practice have grown exponentially in recent years. This article addresses methodological considerations in CBTI validation studies, emphasizing those design issues relevant to customer satisfaction studies. Specifically, issues of response bias are addressed as they relate to selection of raters and test respondents, use of random reports as a "control" for spurious ratings of report validity, and both the format and content of ratings. Deficiencies of various studies from the research literature are reviewed, and advantages and limitations of design alternatives are discussed.Both the availability and adoption of computer-based test interpretation (CBTI) systems have grown exponentially over the last decade. Krug (1984) listed over 200 computer software applications of psychological testing, and most experts predict the growth of computer-based assessment products to continue (Butcher, 1987b;Lanyon, 1987;Mitchell & Kramer, 1985). The potential for CBTIs to facilitate assessment is without question. Research findings generally indicate that "well-designed statistical treatment of test results and ancillary information will yield more valid assessments than will an individual professional using the same information" (American Psychological Association, 1986, p. 13). Computerized interpretive narratives, when developed on a broad actuarial foundation of empirical findings relating test indices to relevant external criteria, offer distinct advantages, including (a) economy of processing and more effective use of professional resources; (b) accuracy and consistency of scoring and implementation of interpretive decision rules; (c) virtually unlimited capacity for storage, indexing, and retrieval of relevant information from the clinical and research literature regarding test-behavior relationships; (d) ability to subject test indicators to complex, configural analyses; and (e) potential for automated collection and analysis of extensive normative data bases (Jackson, 1985;Krug, 1987).However, the proliferation of computer-based assessment has also generated considerable controversy (BersofF, 1989

Criminal personality profiling in the investigation of violent crime: Recent advances and future directions

“…Another area of potential value is the design of so-called "consumer satisfaction" studies, such as those often used in assessing the validity or utility of computerized psychological testing reports (Snyder, Widiger, & Hoover, 1990). Under this research paradigm, resulting profiles can be assessed for their accuracy and validity by those who make use of the profiles, namely criminal investigators and police agencies.…”

Computer-assisted judgment: Defining strengths and liabilities.

“…Because such studies do not rely on independent criterion data regarding the test respondent and therefore are less difficult and less costly to conduct, customer satisfaction studies of CBTI validity are more common than other types of CBTI validity research (although even customer satisfaction studies are infrequent relative to the number of commercial CBTI systems available). D. K. Snyder et al (1990) noted several methodological considerations germane to CBTI customer satisfaction studies. Among these were the need for (a) a relatively large, representative sample of users or potential users of that interpretive system; (b) adequate sampling of test respondents both with respect to common sociodemographic moderators, as well as the behavioral domains addressed by the test; and (c) specific ratings of discrete narrative elements along multiple dimensions such as those noted above.…”

“…Although above-chance accuracy has been found for some instruments (e.g., for the Millon Clinical Multiaxial Inventory by Moreland & Onstad, 1987; and for the Marital Satisfaction Inventory by Hoover & Snyder, 1991), Butcher et al's (2000) review indicated that for CBTIs for other instruments (e.g., the 16 Personality Factor Questionnaire and Rorschach), no differences have been found between accuracy ratings for bona fide and bogus reports. Moreover, D. K. Snyder et al (1990) noted that even studies controlling for Barnum effects may be compromised by a "halo rating" response bias whereby differences in clinicians' ratings between real and bogus reports are artifactually inflated by one obviously invalid element of the bogus narrative. To eliminate or reduce this confound, D. K. Snyder et al argued for interweaving real and bogus interpretive elements within CBTI narratives when controlling for Bamum effects in customer satisfaction studies.…”

Validity and utility of computer-based test interpretation.

“…He also provided several key criticisms of his own with respect to computerized test interpretations, including the interpretations' failure to recognize test takers' uniqueness, the tendency for the interpretations to be unsigned (ostensibly leaving no one directly accountable for the interpretations' contents), and the tendency to be viewed as an end rather than as means to an end. Even more recently, Snyder, Widiger, and Hoover (1990) voiced their own concerns with CBTIs, concluding that "[w]hat the [CBTI] literature lacks (and desperately needs) are rigorously controlled experimental studies examining methodological issues in CBTI validation per se" and recommending specifically that future studies "include representative samples of both [CBTI] consumers and test respondents" and "use characteristics of each as moderator variables in analyzing reports ' generalizability" (p. 476).…”