No abstract

Recent developments in multifaceted Rasch measurement (Linacre, 1989) have made possible new kinds of investigation of aspects (or 'facets') of performance assessments. Relevant characteristics of such facets (for exam ple, the relative harshness of individual raters, the relative difficulty of test tasks) are modelled and reflected in the resulting person ability measures. In addition, bias analyses, that is interactions between elements of any facet, can also be analysed. (For the facet 'person', an element is an individual candidate; for the facet 'rater', an element is an individual judge, and so on.) This permits investigation of the way a particular aspect of the test situation (type of candidate, choice of prompt, etc.) may elicit a consistently biased pattern of responses from a rater. The purpose of the research is to investigate the use of these analytical techniques in rater training for the speaking subtest of the Occupational English Test (OET), a specific-purpose ESL performance test for health professionals. The test involves a role-play based, profession-specific inter action, involving some degree of choice of role-play material. Data are presented from two rater training sessions separated by an 18-month interval and a subsequent operational test administration session. The analysis is used to establish 1) consistency of rater characteristics over different occasions; and 2) rater bias in relation to occasion of rating. The study thus addresses the question of the stability of rater characteristics, which has practical implications in terms of the accreditation of raters and the requirements of data analysis following test administration sessions. It also has research implications concerning the role of multifaceted Rasch measurement in understanding rater behaviour in performance assessment contexts.

)test development efforts. As part of the foundation for the development of the next generation TOEFL test, papers and research reports were commissioned from experts within the fields of measurement, language teaching, and testing through the TOEFL 2000 project. The resulting critical reviews, expert opinions, and research results have helped to inform TOEFL program development efforts with respect to test construct, test user needs, and test delivery. Opinions expressed in these papers are those of the authors and do not necessarily reflect the views or intentions of the TOEFL program.These monographs are also of general scholarly interest, and the TOEFL program is pleased to make them available to colleagues in the fields of language teaching and testing and international student admissions in higher education.The TOEFL 2000 project was a broad effort under which language testing at Educational Testing Service ® (ETS ® ) would evolve into the 21st century. As a first step, the TOEFL program revised the Test of Spoken English™ (TSE ® ) and introduced a computer-based version of the TOEFL test. The revised TSE test, introduced in July 1995, is based on an underlying construct of communicative language ability and represents a process approach to test validation. The computer-based TOEFL test, introduced in 1998, took advantage of new forms of assessment and improved services made possible by computer-based testing, while also moving the program toward its longer-range goals, which included:• the development of a conceptual framework that takes into account models of communicative competence a research program that informs and supports this emerging framework a better understanding of the kinds of information test users need and want from the TOEFL test a better understanding of the technological capabilities for delivery of TOEFL tests into the next century Monographs 16 through 20 were the working papers that laid out the TOEFL 2000 conceptual frameworks with their accompanying research agendas. The initial framework document, Monograph 16, described the process by which the project was to move from identifying the test domain to building an empirically based interpretation of test scores. The subsequent framework documents, Monographs 17-20, extended the conceptual frameworks to the domains of reading, writing, listening, and speaking (both as independent and interdependent domains). These conceptual frameworks guided the research and prototyping studies described in subsequent monographs that resulted in the final test model. The culmination of the TOEFL 2000 project is the next generation TOEFL test that will be released in September 2005.As TOEFL 2000 projects are completed, monographs and research reports will continue to be released and public review of project work invited. TOEFL Program Educational Testing Service iii AbstractThis report documents two coordinated exploratory studies into the nature of oral English-foracademic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine...

Discussions of interaction in second language performance assessment have generally been loosely psychological in orientation, forming part of attempts to model the nature of communicative ability within the individual But in investigating the validity of performance assessments involving interactions between individuals (for example between candidate and interlocutor m speaking assessments), the intrinsically social nature of performance needs to be recognized What would be the consequences for language testing research if it adopted a social perspective on the nature of interaction? The paper explores the necessity for such a reonentation, and suggests areas that would feature in a consequent research agenda