Monitoring Sources of Variability Within the Test of Spoken English Assessment System

Abstract

The purposes of this study were to examine four sources of variability within the Test of Spoken English (TSE®) assessment system, to quantify ranges of variability for each source, to determine the extent to which these sources affect examinee performance, and to highlight aspects of the assessment system that might suggest a need for change. Data obtained from the February and April 1997 TSE scoring sessions were analyzed using Facets (Linacre, 1999a). The analysis showed that, for each of the two TSE administrations, the test usefully separated examinees into eight statistically distinct proficiency levels. The examinee proficiency measures were found to be trustworthy in terms of their precision and stability. It is important to note, though, that the standard error of measurement varies across the score distribution, particularly in the tails of the distribution. The items on the TSE appear to work together; ratings on one item correspond well to ratings on the other items. Yet, none of the items seem to function in a redundant fashion. Ratings on individual items within the test can be meaningfully combined; there is little evidence of psychometric multidimensionality in the two data sets. Consequently, it is appropriate to generate a single summary measure to capture the essence of examinee performance across the 12 items. However, the items differ little in terms of difficulty, thus limiting the instrument's ability to discriminate among levels of proficiency. The TSE rating scale functions as a five‐point scale, and the scale categories are clearly distinguishable. The scale maintains a similar though not identical category structure across all 12 items. Raters differ somewhat in the levels of severity they exercise when they rate examinee performances. The vast majority used the scale in a consistent fashion, though. If examinees' scores were adjusted for differences in rater severity, the scores of two‐thirds of the examinees in these administrations would have differed from their raw score averages by 0.5 to 3.6 raw score points. Such differences can have important consequences for examinees whose scores lie in critical decision‐making regions of the score distribution.

Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments

“…In conclusion, the findings of the present study concur with the previous studies by confirming that raters may be affected by factors other than the actual performance of the test-takers (e.g., Chalhoub-Deville, 1995;Chalhoub-Deville & Wigglesworth, 2005;Lumley & McNamara, 1995;Myford & Wolfe, 2000;Winke & Gass, 2012). Whether random or systematic, similar to the other studies, measurement error was observed in this study underlining the influential factors that may cause disagreement within and/or among the raters' judgments in oral performance assessments.…”

“…In other words, 75 % of the Total Scores assigned by these 15 raters ranked lower or higher in the post-test, ranging from one point difference to more than 10 points. As discussed by Myford and Wolfe (2000), one point may not seem like or be considered as a large difference, but it can have an important effect for the test takers whose scores are around borderline/pass score. Figure 2 below presents the results about the raters' behavior in terms of (a) whether there was a statistically significant difference between their pre-and post-test scores, and (b) whether they referred to the proficiency levels of the students in their think aloud protocols.…”

“…While four main types of rater effects (i.e., halo effects, central tendency, restriction of range, and leniency/severity) are discussed in much detail in the literature (e.g., Lumley & McNamara, 1995;Myford & Wolfe, 2000), the construct-irrelevant factors, the factors other than the actual performances of test-takers that affect raters' behaviors, scoring process, and final scorings, have not been completely explored (Kang, 2012;Stoynoff, 2012). The differences in rater behaviors in terms of leniency/severity toward a particular performance have led researchers to look at another aspect of fair scoring: bias which is an important concept in language testing since test results should be "free from bias" (Weir, 2005, p. 23).…”

“…Previous studies have investigated rater effects on oral test scores from different perspectives such as the raters' educational and professional experience (e.g., Chalhoub-Deville, 1995), raters' nationality and native language (e.g., Chalhoub-Deville & Wigglesworth, 2005;Winke & Gass, 2012;Winke et al, 2011), rater training (e.g., Lumley & McNamara, 1995;Myford & Wolfe, 2000), and the gender of candidates and/or interviewers (e.g., O'Loughlin, 2002;O'Sullivan, 2000). For instance, Lumley and McNamara (1995) examined the effect of rater training on the stability of rater characteristics and rater bias whereas MacIntyre, Noels, and Clément (1997) examined bias in self-ratings in terms of participants' perceived competence in an L2 in relation to their actual competence and language anxiety.…”

Investigating Assessor Effects in National Board for Professional Teaching Standards Assessments for Early Childhood/Generalist and Middle Childhood/Generalist Certification

“…The extension of the Rasch model for analyses of assessor-mediated ratings is called the many-faceted Rasch model (FACETS model, Linacre, 1989). The FACETS model has been used to examine the psychometric quality of a variety of performance assessments based on assessor-mediated ratings (e.g., Engelhard, 1992Engelhard, , 1994Engelhard, , 1996Heller, Sheingold, & Myford, 1998;Linacre, Engelhard, Tatum, & Myford, 1994;Lunz & Stahl, 1990;Lunz, Wright, & Linacre, 1990;Myford, Marr, & Linacre, 1996;Myford & Mislevy, 1995;Myford & Wolfe, 2000a;Paulukonis, Myford, & Heller, 2000;Wolfe, Chiu, & Myford, 1999). It should be stressed that the FACETS model provides additional information that supplements, rather than supplants, the inferences provided by more traditional methods that have been used previously to analyze the NBPTS assessments.…”