Multi Facet Rasch Analysis in tests of oral production

Abstract

The difficulty in constructing tests of oral production lies with the rating schemes and assessment frameworks, their construction and validation and subsequently application. This paper advocates extensive use of numerical data and statistical procedures alongside qualitative and intuitive methodologies, where the common denominator lies in the fact that all of the empirical measures involved make use of some form of a performance model allowing to make predictions about the examinee behaviour in order to verify goodness of fit of the observable rating data.

Multi Facet Rasch Analysis in tests of oral production

“…The planning and specification stage is crucial in a number of ways to the viability of the examination. The issues connected with the standarisation and verification pertaining to reliability and validity in connection with more general considerations in language testing, exam development and evaluation, construct development and operationalisation have been presented in numerous texts in the field (Luoma, 2004;Bond & Fox, 2007;Fox, Wesche & Bayliss, 2007;Fulcher & Davidson, 2007;Douglas, 2009;Krakowian, 2010Krakowian, , 2011Taylor, 2011;Fulcher, 2013;Fulcher, 2014;Leong, Bartram & Iliescu, 2016).…”

“…Numerous programs and procedures exist that are capable of performing such analyses (cf. Krakowian, 2010Krakowian, , 2011. BIGSTEPS and FACETS, are by far the best known, and what is more, now feely available in the original DOS based versions, following the development of newer more user-friendly, and commercially available versions.…”

“…Neither of them separately or together, however, can be exhaustively indicative of raters rating carelessly without paying attention to the true merit of the samples (Wright & Masters, 1982;Bond & Fox, 2007;Salkind, 2007;Kaplan & Saccuzzo, 2009). A freeware program for Multi Facet Rasch Analysis, RarterGrinder, is available at the Institute of English Studies at Łódź University, and its operations are somewhat documented in two studies by Krakowian (2011). The main focus of the program, apart from providing the usual indices of goodness of fit is to explicitly identify the aberrant patterns of behaviour in raters and to indicate, leniency and severity as well as careless ratings.…”