Automated Tools for Subject Matter Expert Evaluation of Automated Scoring

Abstract

As automated scoring of complex constructed-response examinations reaches operational status, the process of evaluating the quality of resultant scores, particularly in contrast to scores of expert human graders, becomes as complex as the data itself. Using a vignette from the Architectural Registration Examination (ARE), this paper explores the potential utility of classification and regression trees (CART) and Kohonen self-organizing maps (SOM) as tools to facilitate subject matter expert (SME) examination of the fine-grained (feature level) quality of automated scores for complex data, with implications for the validity of the resultant scores. The paper explores both supervised and unsupervised learning techniques, the former being represented by CART (Breiman, Friedman, Olshen, & Stone, 1984) and the latter by SOM (Kohonen, 1989). Three applications comprise this investigation, the first of which suggests that CART can facilitate efficient and economical identification of specific elements of complex solutions that contribute to automated and human score discrepancies. The second application builds on the first by exploring CART's value for efficiently and accurately automating case selection for human intervention to ensure score validity. The final application explores the potential for SOM to reduce the need for SMEs in evaluating automated scoring. While both supervised and unsupervised methodologies examined were found to be promising tools for facilitating SME roles in maintaining and improving the quality of automated scoring, such applications remain unproven and further studies are necessary to establish the reliability of these techniques.

Understanding Mean Score Differences Between the e‐rater® Automated Scoring Engine and Humans for Demographically Based Groups in the GRE® General Test

“…CART (Breiman, Friedman, Olshen, & Stone, ) has been used previously in the context of automated scoring by Zechner, Higgins, Xi, and Williamson () for building and evaluating scoring models for the SpeechRater sm automated scoring service and by Williamson, Bejar, and Sax () as an automated tool to help subject matter experts (SMEs) evaluate the human and machine score discrepancies. As Williamson et al () noted, CART has been successfully used in prior research on classification problems in psychometrics (e.g., Sheehan, 1997, as cited in Williamson et al, , for proficiency scaling and diagnostic assessment; Bejar, Yepes‐Baraya, & Miller, 1997, as cited in Williamson et al, , for modeling rater cognition; Holland, Ponte, Crane, & Malberg, 1998, as cited in Williamson et al, , for computerized adaptive testing).…”