EFFECT OF IMMEDIATE FEEDBACK AND REVISION ON PSYCHOMETRIC PROPERTIES OF OPEN‐ENDED GRE^® SUBJECT TEST ITEMS

Abstract

Registered examinees for the GRE® Subject Tests in Biology and Psychology participated in a Web‐based experiment where they answered open‐ended questions that required a short answer of 1‐3 sentences. Responses were automatically scored by natural language processing methods (the c‐rater™ scoring engine) immediately after participants submitted their responses. Based on natural language processing methods (the c‐rater scoring engine), participants received immediate feedback on the correctness of their answers and an opportunity to revise their answers. A significant revision effect was found. Participants were able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of revised scores was higher than initial scores, although the correlations of the initial and revised scores with the GRE Subject Test scores were similar.

Automated Writing Analysis for Writing Pedagogy

“…AWE programs are promoted as effective enhancers of process writing instruction, compelling guides for student revision, and robust vehicles of consistent writing and evaluation across the curriculum. They are also presumed to motivate multiple drafting and revision, foster learner autonomy, and enhance the instructional dynamic by supporting the drive toward individualized instruction (Attali & Powers, 2008;Burstein, 2012).…”

Automated Scoring of Constructed‐Response Science Items: Prospects and Obstacles

“…Automated scoring has been widely applied in educational research to improve scoring efficiency and shorten the time between test administration and when teachers, test takers, and score users receive test results. Research on automated scoring has covered domains such as writing quality (Burstein & Marcu, ; Foltz, Laham, & Landauer, ), mathematics (Bennett & Sebrechts, ; Sandene, Horkay, Bennett, Braswell, & Oranje, ), written content (Attali & Powers, ; Dzikovska, Nielsen, & Brew, ; Graesser, ; Leacock & Chodorow, ; Mitchell, Russell, Broomhead, & Aldridge, ; Nielsen, Ward, & Martin, ; Sukkarieh & Bolge, ), speech (Bernstein, Van Moere, & Cheng, ; Higgins, Zechner, Xi, & Williamson, ), and other education related topics (Sargeant, Wood, & Anderson, ).…”

Validation of automated scoring of science assessments

“…Over the last two decades, automated scoring has been widely developed and used in a variety of content domains, such as mathematics (Bennett & Sebrechts, ; Sandene, Horkay, Bennett, Braswell, & Oranje, ), science (Linn et al, ; Liu et al, ; Nehm, Ha, & Mayfield, ), and language testing (Bernstein, Van Moere, & Cheng, ; Higgins, Zechner, Xi, & Williamson, ), to name a few. Furthermore, in assessing written responses across content domains, automated scoring has been used to evaluate rubric dimensions, such as content (Attali & Powers, ; Dzikovska et al, ; Leacock & Chodorow, ; Mitchell, Russell, Broomhead, & Aldridge, ; Nielsen, Ward, & Martin, ; Sukkarieh & Bolge, ) and quality (Burstein & Marcu, ; Foltz, Laham, & Landauer, ). The accuracy of automated scores depends on a number of factors, including the content domain, the complexity of the tasks, the levels of the scoring rubrics, and the number of responses available to build the automated scoring models.…”