Revisiting Carroll's scales

Tiselius, Elisabet

doi:10.1075/ata.xiv.07tis

Cited by 27 publications

(24 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Scales can also be analytic, with a number of sub-scales and associated descriptors targeting each major dimension of interpreting quality for each performance level (e.g., Han, 2015;J. Lee, 2008;Liu, 2013;Tiselius, 2009). Rubric scoring is increasingly used in different assessment contexts including interpreter training (J.…”

Section: Rubric Scoringmentioning

confidence: 99%

“…Rubric scoring is increasingly used in different assessment contexts including interpreter training (J. Lee, 2008;Setton & Dawrant, 2016), professional certification (International School of Linguists, 2020;Liu, 2013;NAATI, 2019), and interpreting research (Han, 2018;Tiselius, 2009). The often-cited benefits are its capability of producing reliable and valid assessment results, and its affordability and practicality (for a critical review, see Han, 2018).…”

Section: Rubric Scoringmentioning

confidence: 99%

See 1 more Smart Citation

Interpreting quality assessment re-imagined: The synergy between human and machine scoring

Han

2021

Interpreting and Society

View full text Add to dashboard Cite

Assessment of interpreting quality is a ubiquitous social practice in the interpreting industry and academia. In this article, we focus on both psychometric and social dimensions of assessment practice, and analyse two major assessment paradigms, namely, human rater scoring and automatic machine scoring. Regarding human scoring, we describe five specific methods, including atomistic scoring, questionnaire-based scoring, multi-methods scoring, rubric scoring, and ranking, and critically analyse their respective strengths and weaknesses. In terms of automatic scoring, we highlight four assessment approaches that have been researched and operationalised in cognate disciplines and interpreting studies, including automatic assessment based on temporal variables, linguistic/surface features, machine translation metrics, and quality estimation methodology. Finally, we problematise the socio-technological tension between these two paradigms and envisage human–machine collaboration to produce psychometrically sound and socially responsible assessment. We hope that this article sparks more scholarly discussion of rater-mediated and automatic assessment of interpreting quality from a psychometric-social perspective.

show abstract

Section: Rubric Scoringmentioning

confidence: 99%

Section: Rubric Scoringmentioning

confidence: 99%

Interpreting quality assessment re-imagined: The synergy between human and machine scoring

Han

2021

Interpreting and Society

View full text Add to dashboard Cite

show abstract

“…For the investigation and the comparison of the human and the machine output, an evaluation framework derived from the Interpreting Studies (Tiselius, 2009) is chosen and slightly adapted. The framework is "assumed to account for central aspects of the interpreted event but not for its entirety as a communicative event" (Tiselius, 2009, p. 99).…”

Section: Evaluation Framework and Proceduresmentioning

confidence: 99%

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

2021

View full text Add to dashboard Cite

The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2021) featured this year four shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Multilingual speech translation, (iv) Low-resource speech translation. A total of 22 teams participated in at least one of the tasks. This paper describes each shared task, data and evaluation metrics, and reports results of the received submissions.

show abstract

“…Using descriptor-based analytic rating scales to assess interpreting quality is gaining currency in interpreter certification testing (Angelelli, 2009; Certification Commission for Healthcare Interpreters (CCHI), 2011; Han, 2015aHan, , 2016aIoL Educational Trust, 2010;Jacobson, 2009;Liu, 2013;Turner et al, 2010;Wu, Liu, & Liao, 2013), in interpreter educational assessment (Bontempo & Hutchinson, 2011;Lee, 2008;Tiselius, 2009;Wang, 2011;Wang, Napier, Goswell, & Carmichael, 2015;Zhao & Dong, 2013) and in interpreting research (Cheung, 2014;Lin, Chang, & Kuo, 2013;McDermid, 2014). This popular trend, however, does not necessarily confirm the utility and functionality of descriptor-based rating scales.…”

Section: Quality Assessment Of Interpreting: Using Analytic Rating Scalesmentioning

confidence: 99%

Using analytic rating scales to assess English/Chinese bi-directional interpretation: A longitudinal Rasch analysis of scale utility and rater behavior

Han

2018

LANS-TTS

View full text Add to dashboard Cite

Descriptor-based analytic rating scales have been increasingly used to assess interpretation quality. However, little empirical evidence is available to unequivocally support the effectiveness of rating scales and rater reliability. This longitudinal study thus attempts to shed insight into scale utility and rater behavior in English/Chinese interpretation performance assessment, using multifaceted Rasch measurement. Specifically, the study focuses on criterion/scale difficulty, scale effectiveness, rater severity/leniency and rater self-consistency between English/Chinese interpreting and over three time points. Research results are discussed, highlighting the utility of analytic rating scales and the variability of rater behavior in interpretation assessment. The results also have implications for developing reliable, valid, and practical instruments to assess interpretation quality.

show abstract

Revisiting Carroll's scales

Cited by 27 publications

References 10 publications

Interpreting quality assessment re-imagined: The synergy between human and machine scoring

Interpreting quality assessment re-imagined: The synergy between human and machine scoring

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

Using analytic rating scales to assess English/Chinese bi-directional interpretation: A longitudinal Rasch analysis of scale utility and rater behavior

Contact Info

Product

Resources

About