Score Dependability in ESL Oral Performance Assessment : A Generalizability Theory Approach

Kim, Younhee

doi:10.18823/asiatefl.2017.14.3.14.564

Cited by 2 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although Chu's [14] research results showed negative washback effects of the EGB on the teachers and students and that the EGB did not fulfill the participating students' needs, this study's results in fact suggested that technology university students, including both English and non-English majors, were uniformly in favor of the establishment of the EGB and believed it would help their future career development. These findings echo some other researchers' (for example, Wu and Lee [3]; Chen and Liu [6]; Min [7]; Kim [8]) results, as discussed above. Most people may see tests as a demotivating factor for foreign language learners and standardized tests might be unfair to students from less advantageous backgrounds, yet most of the students in the present study seemed to regard the EGB as a motivation for them to study English harder.…”

Section: Discussionsupporting

confidence: 91%

“…Establishing the EGB policy has led to a public discourse among the scholars and linguists in Taiwan who have been keen to ascertain the effectiveness of the implementation of the policy [3]. A wide range of studies, including those of Chen and Liu [6], Min [7], and Kim [8], have examined the students' learning motivation and their perceptions regarding the standardization tests prior to, and following, the deployment of the EGB policy. The general finding in the studies is that the policy has been pivotal in the development of positive attitudes and acceptability regarding the learning of English.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Technology University Students’ Attitudes towards the English Graduation Benchmark (EGB): A Comparison between English-major and non-English Major Students

Lai

2020

Sustainability

View full text Add to dashboard Cite

The English Graduation Benchmark (EGB) is a common criterion adopted by many universities in Taiwan, including both comprehensive and technology universities to monitor their graduates’ English ability. However, this common criterion has caused some debates in recent years about its appropriateness and fairness to students from less advantageous backgrounds. Some universities have decided to stop this policy in practice. This study aims to investigate technology university students’ perspectives and attitudes towards the EGB, as well as to compare and contrast the opinions of students from English and non-English departments. A quantitative method was adopted for this study. The questionnaire included: (1) Participants’ background information, (2) their experience of taking the proficiency tests, and (3) their thoughts on the EGB policy. In total, 360 students participated in this study. The results showed that most technology university students agreed with the establishment of the EGB. Students from English and non-English departments had positive attitudes because the benchmark motivated them to learn English in some way. Last but not least, technology university students considered the EGB to be useful for their future career development. This paper considers the possible implications of these findings for universities and the government, and suggests how they might re-conceptualize their policies to make them more sustainable.

show abstract

Section: Discussionsupporting

confidence: 91%

Section: Introductionmentioning

confidence: 99%

Technology University Students’ Attitudes towards the English Graduation Benchmark (EGB): A Comparison between English-major and non-English Major Students

Lai

2020

Sustainability

View full text Add to dashboard Cite

show abstract

“…Test takers' own speaking ability is the main determining factor for the scores they receive; however, the rater factor also accounts for a substantial role in the fluctuation in their scores. Such a finding is in line with that of Kim (2017), who found that the scores awarded to test takers is more dependent to their own true oral ability.…”

Section: Discussionsupporting

confidence: 89%

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

Bijani¹,

Khabiri²

2017

The Journal of AsiaTEFL

View full text Add to dashboard Cite

Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters' severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters' biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.

show abstract

Score Dependability in ESL Oral Performance Assessment : A Generalizability Theory Approach

Cited by 2 publications

References 9 publications

Technology University Students’ Attitudes towards the English Graduation Benchmark (EGB): A Comparison between English-major and non-English Major Students

Technology University Students’ Attitudes towards the English Graduation Benchmark (EGB): A Comparison between English-major and non-English Major Students

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

Contact Info

Product

Resources

About