A methodology for scoring open-ended architectural design problems.

Bejar, Isaac I.

doi:10.1037/0021-9010.76.4.522

Cited by 59 publications

(38 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the context of the TPO assessment, the specific implication is of whether the examinee responses are obtained and scored appropriately so that the resulting score accurately represents the quality of the performance. For the specific question of use of automated scoring, there is a rich precedence of studies comparing the results of automated and human scoring as the basis for evaluating the validity of automated scores from the Evaluation perspective (e.g., Bejar, 1991;Bennett, Sebrechts, & Marc, 1994;Braun, Bennett, Frye, & Soloway, 1990;Clauser et al, 1995;Clauser et al, 1997;Kaplan & Bennett, 1994;Page & Petersen, 1995;Sebrechts, Bennett, & Rock, 1991;Williamson et al, 1999). It is generally acknowledged that human scores are not perfect and therefore may not be an ideal basis for evaluation of the quality of automated scoring, suggesting that the term gold standard may be a misnomer.…”

Section: The Validity Argument For Scoring Tpo With Speechrater V10mentioning

confidence: 99%

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

This report presents the results of a research and development effort for SpeechRater SM Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL ® ) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short-and long-term goals for iterative improvements to SpeechRater scoring. which is used by prospective test takers to prepare for the official TOEFL iBT test. This study reports the development and validation of the system for low-stakes practice purposes. The process we followed to build this system represented a principled approach to maximizing 2 essential qualities: substantively meaningful and technically sound. In developing and evaluating the features and the scoring models to predict human assigned scores, we engaged both content and technical experts actively to ensure the construct representation and technical soundness of the system. We compared primarily two alternative methodologies of building scoring modelsmultiple regression and classification trees-in terms of their construct representation and empirical performance in predicting human scores. Based on the evaluation results, we concluded that a multiple regression model with feature weights determined by content experts was superior to the other competing models evaluated.We then used an argument-based approach to integrate and evaluate the existing evidence to support the use of SpeechRater SM v1.0 in a low-stakes practice environment. The argumentbased approach provided a mechanism for us to articulate the strengths and weaknesses in the validity argument for using SpeechRater v1.0 and put forward a transparent argument for using it for a low-stakes practice environment. In particular, the construct representation of the multiple regression model with expert weights was sufficiently broad to justify its use in a low-stakes application. While some higher-order aspects of the speaking construct (such as content and organization) are missing, more basic aspects of the construct (such as pronunciation and fluency) are richly represented. In addition, these different parts of the speaking construct tend to be highly correlated, so that the absence of higher order factors is not as detrimental to the model's agreement with human raters as it otherwise might be. The model's agreement with human raters was not sufficiently high to support high-stakes decisions but was still suitable for use in low-stakes applications. The correlation of the 6-item aggregate score with human raters was .57 and was deemed acceptable given the lo...

show abstract

Section: The Validity Argument For Scoring Tpo With Speechrater V10mentioning

confidence: 99%

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…Appearing in this decade were ETS's first attempts at automated scoring, including of computer science subroutines (Braun et al 1990), architectural designs (Bejar 1991), mathematical step-by-step solutions and expressions (Bennett et al 1997;Sebrechts et al 1991), short-text responses (Kaplan 1992), and essays (Kaplan et al 1995). By the middle of the decade, the work on scoring architectural designs had been implemented operationally as part of the National Council of Architectural Registration Board's Architect Registration Examination (Bejar and Braun 1999).…”

Section: Constructed-response Formats and Performance Assessmentmentioning

confidence: 99%

Advancing Human Assessment: A Synthesis Over Seven Decades

Bennett

Davier

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This book has documented the history of ETS's contributions to educational research and policy analysis, psychology, and psychometrics. We close the volume with a brief synthesis in which we try to make more general meaning from the diverse directions that characterized almost 70 years of work.Synthesizing the breadth and depth of the topics covered over that time period is not simple. One way to view the work is across time. Many of the book's chapters presented chronologies, allowing the reader to follow the path of a research stream over the years. Less evident from these separate chronologies was the extent to which multiple streams of work not only coexisted but sometimes interacted.From its inception, ETS was rooted in Henry Chauncey's vision of describing individuals through broad assessment of their capabilities, helping them to grow and society to benefit (Elliot 2014). Chauncey's conception of broad assessment of capability required a diverse research agenda.Following that vision, his research managers assembled an enormous range of staff expertise. Only through the assemblage of such expertise could one bring diverse perspectives and frameworks from many fields to a problem, leading to novel solutions.In the following sections, we summarize some of the key research streams evident in different time periods, where each period corresponds to roughly a decade. Whereas the segmentation of these time periods is arbitrary, it does give a general This work was conducted while M. von Davier was employed with Educational Testing Service.

show abstract

“…Of course, automated scoring is straightforward for multiple-choice tests, but it has also been used to assess complex skills in a variety of domains, such as architectural design (e.g., Bejar, 1991) and physician patient management (e.g., Clauser et al, 1997). To use automated scoring, tests would need to be administered on computer systems (which also promotes scalability of test administration).…”

Section: Develop and Implement Instruments To Assess Proficiencymentioning

confidence: 99%