An Examination of Rater Orientations and Test-Taker Performance on English-for-Academic-Purposes Speaking Tasks

Abstract

)test development efforts. As part of the foundation for the development of the next generation TOEFL test, papers and research reports were commissioned from experts within the fields of measurement, language teaching, and testing through the TOEFL 2000 project. The resulting critical reviews, expert opinions, and research results have helped to inform TOEFL program development efforts with respect to test construct, test user needs, and test delivery. Opinions expressed in these papers are those of the authors and do not necessarily reflect the views or intentions of the TOEFL program.These monographs are also of general scholarly interest, and the TOEFL program is pleased to make them available to colleagues in the fields of language teaching and testing and international student admissions in higher education.The TOEFL 2000 project was a broad effort under which language testing at Educational Testing Service ® (ETS ® ) would evolve into the 21st century. As a first step, the TOEFL program revised the Test of Spoken English™ (TSE ® ) and introduced a computer-based version of the TOEFL test. The revised TSE test, introduced in July 1995, is based on an underlying construct of communicative language ability and represents a process approach to test validation. The computer-based TOEFL test, introduced in 1998, took advantage of new forms of assessment and improved services made possible by computer-based testing, while also moving the program toward its longer-range goals, which included:• the development of a conceptual framework that takes into account models of communicative competence a research program that informs and supports this emerging framework a better understanding of the kinds of information test users need and want from the TOEFL test a better understanding of the technological capabilities for delivery of TOEFL tests into the next century Monographs 16 through 20 were the working papers that laid out the TOEFL 2000 conceptual frameworks with their accompanying research agendas. The initial framework document, Monograph 16, described the process by which the project was to move from identifying the test domain to building an empirically based interpretation of test scores. The subsequent framework documents, Monographs 17-20, extended the conceptual frameworks to the domains of reading, writing, listening, and speaking (both as independent and interdependent domains). These conceptual frameworks guided the research and prototyping studies described in subsequent monographs that resulted in the final test model. The culmination of the TOEFL 2000 project is the next generation TOEFL test that will be released in September 2005.As TOEFL 2000 projects are completed, monographs and research reports will continue to be released and public review of project work invited. TOEFL Program Educational Testing Service iii AbstractThis report documents two coordinated exploratory studies into the nature of oral English-foracademic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine...

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATERSM V1.0

“…Based on Brown, Iwashita, & McNamara (2005), the rubrics for the TOEFL iBT Speaking test were reflective of what teachers of English as a second language and applied linguists thought were important in evaluating candidates' speaking performance in an academic environment. However, the features used in the automated scoring model were only a subset of the criteria used by the human raters, reducing the model's power in explaining candidates' performance on realworld speaking tasks.…”

How Do Raters Judge Spoken Vocabulary?

“…In particular, descrepancy between multiple raters' judgement has always been an area of intest since 1920s. Some interesting findings relevant to vocabulary assessment in oral examinations have been identified in the past studies (Brown, Iwashita, & McNamara, 2005;Lorenzo-Dus & Meara, 2005;Malvern & Richards, 2002;Read, 2000Read, , 2005. To date, however, few studies have focused solely on rater performance in assessing vocabulary.…”

“…Some findings include: (1) Raters' judgements on vocabulary do not correlate with lexical diversity (D) in oral proficiency interviews (Lorenzo-Dus & Meara, 2005;Malvern & Richards, 2002); (2) Raters exhibit idiosyncratic approaches regarding saliency of lexical features in assessing vocabulary in oral interviews. They typically make more negative than positive comments on vocabulary (Brown, 2006); (3) Raters' judgements are sensitive to word types, tokens, and difficult words in OPIs (Brown et al, 2005;Lorenzo-Dus & Meara, 2005); (4) Raters have conflicting views on assessing linguistic aspects vis-à-vis pragmatic aspects of vocabulary in oral examinations (Brown et al, 2005); (5) It is difficult for raters to assess vocabulary at adjacent IELTS band levels (Read, 2005); (6) High correlations have been found between subcategories in oral examinations, such as vocabulary, grammar, fluency, etc. (Brown and Taylor, 2006;Malvern and Richards, 2002;Taylor and Jones, 2001); and (7) Vocabulary and grammar were prone to be rated more harshly than other constructs in oral examinations (Galaczi, 2005).…”

The Diversity in an English Oral Proficiency Test

“…has assumed that different ratings can be controlled by rater training with explicit assessment criteria and samples of performance at different levels [8].…”

“…English language oral proficiency is usually evaluated by human raters, mostly native speakers [8]. Raters play a major role in the assessment process and influence the quality and meaning of scores obtained.…”