Investigating variability in tasks and rater judgements in a performance test of foreign language speaking

Abstract

Much of the recent debate that has surrounded the development and use of 'performance', or 'communicative' language tests has focused on a supposed trade-off between two sets of desirable qualities: correspondence between test tasks and test performance to nontest language use for content relevance; and reliability of scores derived from test performance. One area that has been of particular concern with performance tests is the potential variability in tasks and rater judgements, and this has been investigated in the language testing literature with two complementary approaches: generalizability the ory and many faceted Rasch modelling. GENOVA, which performs general izability theory analyses, estimates the relative contribution of variation in test tasks and rater judgements to variation in test scores. FACETS, which performs many faceted Rasch modelling, estimates differences in task difficulty and rater severity, and adjusts ability estimates of test takers, taking these differences into account. In this article we first discuss the design and development of a foreign language (Spanish) test battery that was designed for two purposes: first, to place University of California Education Abroad students into programmes at universities abroad that are appropriate for their level of language ability, and secondly to provide diagnostic information that will be useful for designing appropriate teaching and learning pro grammes for prospective education abroad students. The test battery consists of four subtests: reading, listening and note-taking, speaking, and writing. All subtests share a common theme or topic, and are interdependent. We then discuss the results of the GENOVA and FACETS analyses of the speaking subtest, based on a full field trial with a group of University of California undergraduate students who had been selected for participation in the Education Abroad Program. Finally, we discuss the implications of these results for the use of G-theory and many faceted Rasch modelling for the development of performance tests of foreign language ability.

Score Dependability in ESL Oral Performance Assessment : A Generalizability Theory Approach

“…The two univariate G-study designs showed that the largest variance component was in the test-takers themselves, indicating that most variability in ESL oral performance can be explained by test-takers' true ability, not by construct-irrelevant factors such as tasks or raters. The contribution of the interaction between test-takers and raters to the total score variance estimate was found to be substantial (approximately 5% to 6%), confirming the findings of Bachman, Lynch, and Mason (1995) and Lynch and McNamara (1998). This suggests that ESL teachers exhibited somewhat different severity patterns to a certain group of ESL learners.…”

A Study on the Validity of Vstep Writing Tests for the Sake of Regional and International Integration

“…Therefore, this study adopts the triangulation approach employed in other studies (e.g., Bachman, Lynch, & Mason, 1995;Lynch & McNamara, 1998) of combining manyfacet Rasch measurement with generalizability theory, while adding consideration of interrater score correlations as an additional source of information on scoring consistency.…”

ESL students’ oral performance in English language school-based assessment: results of an empirical study

“…It was thus apparent that Skehan and his colleagues' (Foster and Skehan 1996;Skehan and Foster 1997) observation that more interactive tasks lead to more complex language performance did not find support in the Bygate and Michel et al (2007) studies. In language testing contexts, a few studies (e.g., Fulcher 1996;Bachman et al 1995) reported significant but small differences in test scores across different types of test tasks. More recently, a number of studies conducted in experimental language testing settings that replicated Skehan's or Robinson's framework concerning the impact of task performance conditions on task performance revealed results that did not lend much support to either of the their theoretical frameworks.…”