Computer‐based Test Interpretations: Some Development and Application Issues

Harris, William G.

doi:10.1111/j.1464-0597.1987.tb01189.x

Cited by 7 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We would not contend that the available methodologies for validating CBTIs are woefully inadequate (cf. Harris, 1987), only that the modal designs used until now have been strikingly deficient. Nor would we suggest, as has Lanyon (1987), that the validity findings in carefully designed studies are likely to be little better than those currently available.…”

Section: Discussionmentioning

confidence: 99%

“…Whether conducting external criterion-or customer-satisfac-tion studies, clear trade-offs exist in pursuing "expert" versus "representative" raters. Given limited resources for conducting a validational study, it seems likely that the extent of training recommended for raters in both the interpretive system and the criterion-rating format (Harris, 1987) and analyses of interrater, intrarater, and temporal reliability suggested by Moreland (1987) would necessarily restrict the number of expert raters included in a study. The breadth of sampling of both raters and respondents contributes to the degree to which findings are relatively free of inherent response biases.…”

Section: Selection Of Cbti Ratersmentioning

confidence: 99%

“…CBTI systems may at times offer only the appearance of objectivity and empirical validity, and the economic gain to the consumer and producer may compromise attention to empirical validation. Harris (1987) noted the paucity of published literature demonstrating the validity of CBTI systems and concluded that “methodologies for validating the accuracy of narrative reports remain woefully inadequate” (p. 240). Similar sentiments have been expressed by Matarazzo (1985), asserting that the validity of computerized narratives constitutes “little more than pseudoscience masquerading as science” (p. 236).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Methodological considerations in validating computer-based test interpretations: Controlling for response bias.

Snyder

Widiger²,

Hoover³

1990

Psychological Assessment: A Journal of Consulting and Clinical

View full text Add to dashboard Cite

Although computer-based test interpretation (CBTI) systems have been operational for nearly 25 years, their availability and adoption in routine clinical practice have grown exponentially in recent years. This article addresses methodological considerations in CBTI validation studies, emphasizing those design issues relevant to customer satisfaction studies. Specifically, issues of response bias are addressed as they relate to selection of raters and test respondents, use of random reports as a "control" for spurious ratings of report validity, and both the format and content of ratings. Deficiencies of various studies from the research literature are reviewed, and advantages and limitations of design alternatives are discussed.Both the availability and adoption of computer-based test interpretation (CBTI) systems have grown exponentially over the last decade. Krug (1984) listed over 200 computer software applications of psychological testing, and most experts predict the growth of computer-based assessment products to continue (Butcher, 1987b;Lanyon, 1987;Mitchell & Kramer, 1985). The potential for CBTIs to facilitate assessment is without question. Research findings generally indicate that "well-designed statistical treatment of test results and ancillary information will yield more valid assessments than will an individual professional using the same information" (American Psychological Association, 1986, p. 13). Computerized interpretive narratives, when developed on a broad actuarial foundation of empirical findings relating test indices to relevant external criteria, offer distinct advantages, including (a) economy of processing and more effective use of professional resources; (b) accuracy and consistency of scoring and implementation of interpretive decision rules; (c) virtually unlimited capacity for storage, indexing, and retrieval of relevant information from the clinical and research literature regarding test-behavior relationships; (d) ability to subject test indicators to complex, configural analyses; and (e) potential for automated collection and analysis of extensive normative data bases (Jackson, 1985;Krug, 1987).However, the proliferation of computer-based assessment has also generated considerable controversy (BersofF, 1989

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Selection Of Cbti Ratersmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Methodological considerations in validating computer-based test interpretations: Controlling for response bias.

Snyder

Widiger²,

Hoover³

1990

Psychological Assessment: A Journal of Consulting and Clinical

View full text Add to dashboard Cite

show abstract

“…1956) on ratings of perceived accuracy. As Butcher et al (2000) noted, various studies have shown that consumers assign greater accuracy to narratives with a higher saturation of nonspecific statements, particularly when these descriptions are attributed to persons of authority or expertise (Harris, 1987; C. R. Snyder, Shenkel, & Lowery, 1977). Given this tendency, findings from the international study of computer-based MMPI-2 reports by Butcher et al (1998) might be regarded as less than compelling, a limitation also suggested by Garb (2000b).…”

Section: Validity Of Cbtismentioning

confidence: 99%

Computer-assisted judgment: Defining strengths and liabilities.

Snyder¹

2000

Psychological Assessment

View full text Add to dashboard Cite

Clinicians often fail to recognize limitations in their own subjective judgments, make use of well-developed mechanical-prediction methods, or carefully evaluate which computer-based aids warrant their consideration. This article addresses issues regarding computer-based test interpretations (CBTIs) and computer-based decision making. Comments highlight conclusions reached by other contributors to this Special Section, additional literature bearing on these observations, and implications for consumers of computer-assisted techniques and researchers developing or evaluating these methods. The future of computer-assisted assessment depends on educating clinicians and researchers to be better consumers of existing as well as emerging technologies in this domain.

show abstract

“…Computer-based test interpretations (CBTI's) have dramatically influenced psychological testing in the past 25 years. However, despite the widespread availability of CBTI's, validity studies have been comparatively scarce (Lanyon, 1984; Matarazzo, 1986; Moreland, 1987) and of dubious quality (Harris, 1987; Lanyon, 1987). Furthermore, many available CBTI systems are based on test instruments that themselves are unvalidated (Fowler, 1987).…”

mentioning

confidence: 99%

Validity of the computerized interpretive report for the Marital Satisfaction Inventory: A customer satisfaction study.

Hoover¹,

Snyder²

1991

Psychological Assessment: A Journal of Consulting and Clinical

View full text Add to dashboard Cite

This study examined the validity and clinical utility of the computerized interpretive report for the Marital Satisfaction Inventory (MSI), as rated by a national sampleof 60 marital therapists. Individual accuracy and utility ratings were obtained for separate narrative sections of the computerized report in which Barnum effects were controlled by comparing bona fide interpretive paragraphs with bogus narratives. In addition, "halo" effects were controlled by interweaving bogus and bona fide interpretive sections across alternate versions of "experimental" reports. Clinicians also provided ratings of accuracy and usefulness for final, bona fide reports evaluated in their entirety. Overall, experienced clinicians rated the MSI computerized report favorably in its accuracy, inclusion of relevant information, conciseness and organization, and usefulness in treatment planning.

show abstract

Computer‐based Test Interpretations: Some Development and Application Issues

Cited by 7 publications

References 23 publications

Methodological considerations in validating computer-based test interpretations: Controlling for response bias.

Methodological considerations in validating computer-based test interpretations: Controlling for response bias.

Computer-assisted judgment: Defining strengths and liabilities.

Validity of the computerized interpretive report for the Marital Satisfaction Inventory: A customer satisfaction study.

Contact Info

Product

Resources

About