The Relationship Between Raters' Prior Language Study and the Evaluation of Foreign Language Speech Samples

Winke, Paula; Gass, Susan M.; Myford, Carol M.

doi:10.1002/j.2333-8504.2011.tb02266.x

Cited by 18 publications

(19 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This finding seems to support Hsieh's suggestion that the differences she found between professional and non-professional raters are due to differences in prior exposure to L2 speech rather than to their professional field (Hsieh, 2011, p. 63), thus supporting findings of earlier studies regarding familiarity with L2-accented speech (Derwing & Munro, 1997;Gass & Varonis, 1984;Rubin, 1992;Winke, Gass, & Myford, 2011). However, in our study both rater groups maintained abundant contact with L2 speakers, and therefore exposure to L2 speech does not seem to be a likely explanation.…”

Section: Research Questionssupporting

confidence: 84%

Professional and non-professional raters’ responsiveness to fluency and accuracy in L2 speech: An experimental approach

2017

View full text Add to dashboard Cite

It is general practice to use rater judgments in speaking proficiency testing. However, it has been shown that raters' knowledge and experience may influence their ratings, both in terms of leniency and varied focus on different aspects of speech.The purpose of this study is to identify raters' relative responsiveness to fluency and linguistic accuracy in an occupational context, and to investigate whether professional and non-professional raters with a broad exposure to L2 speech demonstrate similar responsiveness to these two aspects. To this end, an experimental approach was applied. Fluency and accuracy were separated and systematically manipulated. As it is known that foreign accentedness of speech influences raters' judgments, this factor was accounted for. Seventeen responses to a Dutch L2 exam in a vocational context were converted into four different versions manipulated for morphosyntactical accuracy and/or fluency, and read by a Dutch L2 actor, resulting in 68 stimuli. Fifty-five professional raters and 41 non-trained, potential stakeholders holistically rated all stimuli. All raters had extensive prior exposure to spoken L2 Dutch.Linear mixed modeling showed that improvement of either fluency or accuracy led to significantly higher ratings by both linguistically trained and non-trained raters. This finding confirms that both groups perceive these aspects to be important features of speaking proficiency. Raters seemed research-article2017Article 502 Language Testing 35(4)to be more responsive to improvement of accuracy than of fluency. The linguistically non-trained raters seemed to appreciate the fluency improvement more than linguistically trained raters. The linguistically trained raters rewarded morpho-syntactical improvement relatively higher than the non-trained raters. This latter effect was explained by the finding that the linguistically trained raters seemed to be more preoccupied with accuracy, according to their responses to a questionnaire. This result suggests that raters with linguistic expertise were more attentive to accuracy whereas non-trained raters were relatively more attentive to fluency.

show abstract

Section: Research Questionssupporting

confidence: 84%

Professional and non-professional raters’ responsiveness to fluency and accuracy in L2 speech: An experimental approach

2017

View full text Add to dashboard Cite

show abstract

“…Winke et al, 2011. The most informative and important piece of output from Facets analyses is the variable map, which summarizes the key information of each facet and grouping facet into one figure.…”

Section: Findings and Discussionmentioning

confidence: 99%

Second Language Pronunciation Assessment

Fulcher¹,

Browne²

2016

View full text Add to dashboard Cite

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. Full terms of use are available: http://www.bristol.ac.uk/pure/about/ebr-terms This series brings together titles dealing with a variety of aspects of language acquisition and processing in situations where a language or languages other than the native language is involved. Second language is thus interpreted in its broadest possible sense. The volumes included in the series all offer in their different ways, on the one hand, exposition and discussion of empirical findings and, on the other, some degree of theoretical reflection. In this latter connection, no particular theoretical stance is privileged in the series; nor is any relevant perspective -sociolinguistic, psycholinguistic, neurolinguistic, etc. -deemed out of place. The intended readership of the series includes final-year undergraduates working on second language acquisition projects, postgraduate students involved in second language acquisition research, and researchers, teachers and policy-makers in general whose interests include a second language acquisition component. Second Language Pronunciation Assessment SECOND LANGUAGE ACQUISITIONFull details of all the books in this series and of all our other publications can be found on http://www.multilingual-matters.com, or by writing to Multilingual Matters, St Nicholas House, 31-34 High Street, Bristol BS1 2AW, UK. SECOND LANGUAGE ACQUISITION: 107 Second Language Pronunciation AssessmentInterdisciplinary Perspectives All rights reserved. No part of this work may be reproduced in any form or by any means without permission in writing from the publisher. Edited by Talia Isaacs and Pavel Trofi movich MULTILINGUAL MATTERSThe policy of Multilingual Matters/Channel View Publications is to use papers that are natural, renewable and recyclable products, made from wood grown in sustainable forests. In the manufacturing process of our books, and to further support our policy, preference is given to printers that have FSC and PEFC Chain of Custody certification. The FSC and/or PEFC logos will appear on those books where full certification has been granted to the printer concerned.Typeset by Nova Techset Private Limited, Bengaluru and Chennai, India. Printed and bound in the UK by the CPI Books Group Ltd. Printed and bound in the US by Edwards Brothers Malloy, Inc. In Memory of Alan Davies and Danielle Guénette AcknowledgementsThis edited volume, which brings together different but complementary research perspectives to establish a common platform in which to discuss issues relevant to assessing second language (L2) pronunciation, would not have been possible without the contributions and commitment of the authors, who explore key issues through different disciplinary lenses in the chapters that make up this volume. The vision for the book arose during a cold Canadian winter at the beginning of the second decade of the 21st century, when a sense of momentum for i...

show abstract

“…Yet, sometimes even though the rubrics used are appropriate for the goals of the tests, raters may behave differently both in their own scoring processes and from each other while conducting the interviews, interacting with the test-takers and assessing the test-takers' performances. As a result, if raters are affected by some construct-irrelevant factors during the rating process, it is highly possible that they can misjudge the performance of test-takers which can lead to the misinterpretation of scores (Winke, Gass & Myford, 2011). In other words, rater measurement error, that is, "the variance in scores on a test that is not directly related to the purpose of the test" (Brown, 1996, p.188), can result in a lower score than a test-taker really deserves, which in some cases even lead to failing a test.…”

Section: Introductionmentioning

confidence: 99%

“…Previous studies have investigated rater effects on oral test scores from different perspectives such as the raters' educational and professional experience (e.g., Chalhoub-Deville, 1995), raters' nationality and native language (e.g., Chalhoub-Deville & Wigglesworth, 2005;Winke & Gass, 2012;Winke et al, 2011), rater training (e.g., Lumley & McNamara, 1995;Myford & Wolfe, 2000), and the gender of candidates and/or interviewers (e.g., O'Loughlin, 2002;O'Sullivan, 2000). For instance, Lumley and McNamara (1995) examined the effect of rater training on the stability of rater characteristics and rater bias whereas MacIntyre, Noels, and Clément (1997) examined bias in self-ratings in terms of participants' perceived competence in an L2 in relation to their actual competence and language anxiety.…”

Section: Introductionmentioning

confidence: 99%

“…As discussed above, during the rating process, if raters are affected by some factors other than the actual performances of test-takers, it is highly possible that they can misjudge the performance of test-takers which can lead to the misinterpretation of scores (Winke et al, 2011). Moreover, given that paired oral interviews are also widely used in educational settings to assess learners' spoken proficiency, due to the performance-irrelevant factors, a student can get a lower score than he/she deserves, or even worse, fail in the test.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments

Tanriverdi-Köksal¹

2017

HUJE

View full text Add to dashboard Cite

There has been an ongoing debate on the reliability of oral exam scores due to the existence of human raters and the factors that might account for differences in their scorings. This quasi-experimental study investigated the possible effect(s) of the raters' prior knowledge of students' proficiency levels on rater scorings in oral interview assessments. The study was carried out in a pre-and post-test design with 15 EFL instructors who performed as raters in oral assessments at a Turkish state university. In both pre-and post-tests, the raters assigned scores to the same video-recorded oral interview performances of 12 students from three different proficiency levels. While rating the performances, the raters also provided verbal reports about their thought processes. The raters were not informed about the students' proficiency levels in the pre-test, while this information was provided in the posttest. According to the findings, majority of the Total Scores ranked lower or higher in the post-test. The thematic analysis of the raters' video recorded verbal reports revealed that most of the raters referred to the proficiency levels of the students while assigning scores in the post-test. The findings of the study suggest that besides factors such as accent, nationality, and gender of the test-takers and the assessors, raters' prior knowledge of students' proficiency levels could be a variable that needs to be controlled for more reliable test results.Keywords: rater effects, intra-rater reliability, paired oral exams, think-aloud protocols ÖZ: Yaygın olarak kullanılmakta olsa da notlandıran olarak insan faktörünün varlığı ve notlardaki farklılığa neden olan etmenler sebebiyle konuşma sınav notlarının güvenirliği konusunda süregelen bir tartışma vardır. Bu yarı deneysel çalışma, konuşma sınavlarının değerlendirilmesinde, not verenlerin öğrencilerin dil yeterlilik seviyelerini önceden biliyor olmasının verdikleri notları üzerindeki etkilerini araştırmayı amaçlamaktadır. Bu çalışma, Türkiye'deki bir devlet üniversitesinde yabancı dil olarak İngilizce öğreten ve aynı üniversitede konuşma sınavlarında notlandıran olarak görev alan 15 okutman ile ön ve son test olarak iki oturumda yürütülmüştür. Hem ön hem de son testte, notlandıranlar üç farklı seviyeden 12 öğrencinin video kaydına alınmış aynı konuşma sınavı performansları için not vermiştir. Aynı zamanda, performanslar için not verirken, notlandıranlar eş zamanlı olarak ne düşündükleri ile ilgili sözlü bildirimde bulunmuştur. Öğrencilerin dil yeterlilik seviyeleri ile ilgili ön testte herhangi bir bilgi verilmezken, notlandıranlar öğrencilerin seviyeleri konusunda son testte sözlü ve yazılı olarak bilgilendirilmiştir. Sonuçlara göre, önteste kıyasla son testte Toplam Notların büyük çoğunluğunun son testte düştüğü veya yükseldiği saptanmıştır. Tüm notlandıranların video kayıtlı sözlü bildirimleri tematik olarak incelendiğinde, notlandıranların çoğunun son testte not verirken öğrencilerin dil yeterlilik seviyelerine değindikleri gözlemlenmiştir. Çalışmanın ...

show abstract

The Relationship Between Raters' Prior Language Study and the Evaluation of Foreign Language Speech Samples

Cited by 18 publications

References 73 publications

Professional and non-professional raters’ responsiveness to fluency and accuracy in L2 speech: An experimental approach

Professional and non-professional raters’ responsiveness to fluency and accuracy in L2 speech: An experimental approach

Second Language Pronunciation Assessment

Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments

Contact Info

Product

Resources

About