Questing for Quality Estimation A User Study

Escartín, Carla Parra; Béchara, Hanna; Orăsan, Constantin

doi:10.1515/pralin-2017-0032

Cited by 9 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Similarly for the sentence level, Turchi et al (2015) use binary color-coded labels for visualizing MT quality information and assess whether this can lead to noticeable gains in translators' productivity: indeed, the authors show a statistically significant PE speedup, however, only for source sentences containing 5-20 words and a MT quality of more than 0.1 HTER. Inspired by the work of Turchi et al (2015), Parra Escartín et al (2017 modified the Post-Editing Tool (PET, Aziz et al (2012)) to present translators with a traffic light system of sentence-level QE visualizations that indicates whether they need to translate the source text from scratch or post-edit the MT. Their results indicate that good and accurate MT QE, is vital to the efficiency of the translation workflow, and can cut translation time and effort significantly.…”

Section: Related Workmentioning

confidence: 99%

Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output

Shenoy¹,

Herbig²,

Krüger³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Compared to fully manual translation, postediting (PE) machine translation (MT) output can save time and reduce errors. Automatic word-level quality estimation (QE) aims to predict the correctness of words in MT output and holds great promise to aid PE by flagging problematic output. Quality of QE is crucial, as incorrect QE might lead to translators missing errors or wasting time on already correct MT output. Achieving accurate automatic word-level QE is very hard, and it is currently not known (i) at what quality threshold QE is actually beginning to be useful for human PE, and (ii), how to best present wordlevel QE information to translators. In particular, should word-level QE visualization indicate uncertainty of the QE model or not? In this paper, we address both research questions with real and simulated word-level QE, visualizations, and user studies, where time, subjective ratings, and quality of the final translations are assessed. Results show that current wordlevel QE models are not yet good enough to support PE. Instead, quality levels of ≥ 80% F1 are required. For helpful quality levels, a visualization reflecting the uncertainty of the QE model is preferred. Our analysis further shows that speed gains achieved through QE are not merely a result of blindly trusting the QE system, but that the quality of the final translations also improves. The threshold results from the paper establish a quality goal for future wordlevel QE research.

show abstract

Section: Related Workmentioning

confidence: 99%

Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output

Shenoy¹,

Herbig²,

Krüger³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…We find that good MTQE information can improve the post-editing efficiency. This paper builds on our preliminary published work [4] by carrying out a much more detailed analysis of the data.…”

Section: Introductionmentioning

confidence: 99%

The Role of Machine Translation Quality Estimation in the Post-Editing Workflow

Béchara

Orăsan

Escartín³

et al. 2021

Informatics

Self Cite

View full text Add to dashboard Cite

As Machine Translation (MT) becomes increasingly ubiquitous, so does its use in professional translation workflows. However, its proliferation in the translation industry has brought about new challenges in the field of Post-Editing (PE). We are now faced with a need to find effective tools to assess the quality of MT systems to avoid underpayments and mistrust by professional translators. In this scenario, one promising field of study is MT Quality Estimation (MTQE), as this aims to determine the quality of an automatic translation and, indirectly, its degree of post-editing difficulty. However, its impact on the translation workflows and the translators’ cognitive load is still to be fully explored. We report on the results of an impact study engaging professional translators in PE tasks using MTQE. To assess the translators’ cognitive load we measure their productivity both in terms of time and effort (keystrokes) in three different scenarios: translating from scratch, post-editing without using MTQE, and post-editing using MTQE. Our results show that good MTQE information can improve post-editing efficiency and decrease the cognitive load on translators. This is especially true for cases with low MT quality.

show abstract

“…Con respecto a las opciones de implementación descritas, aquellas que consideran la calidad de los segmentos que se proponen al traductor, independientemente de si proceden de una MT o de un sistema de TA, plantean un escenarioóptimo para el uso eficiente de las tecnologías disponibles (Parra Escartín et al, 2017). Por una parte, se facilita la reutilización de segmentos ya traducidos y validados anteriormente, y por otra, se optimiza la tarea de realizar nuevas traducciones, ya sea con la ayuda de propuestas de TA en caso de que resulten adecuadas para una posedición eficiente, ya sea sin ellas, en cuyo caso el traductor formulará su propuesta sin necesidad de trabajar con propuestas de TA de mala calidad.…”

Section: Introductionunclassified

Estrategia multidimensional para la selección de candidatos de traducción automática para posedición

Aranberri¹,

Gibert²

2020

Linguamática

View full text Add to dashboard Cite

Una integración eficiente de un sistema de traducción automática (TA) en un flujo de traducción conlleva la necesidad de distinguir entre oraciones que se benefician de la TA y las que no antes de que pasen a manos del traductor. En este trabajo, cuestionamos el uso por separado de las dimensiones de esfuerzo de posedición de Krings (2001) para clasificar oraciones en aptas para traducir o poseditar al entrenar modelos de predicción y abogamos por una estrategia multidimensional. A partir de una tarea de posedición en un escenario real, se recogen mediciones de los tres parámetros de esfuerzo, a saber, tiempo, tasa de palabras poseditadas, y percepción del esfuerzo, como representativos de las tres dimensiones (temporal, técnica y cognitiva). Los resultados muestran que, a pesar de que existen correlaciones entre las mediciones, los parámetros difieren en la clasificación de un número elevado de oraciones. Concluimos que la estrategia multidimensional es necesaria para estimar el esfuerzo real de posedición.

show abstract

Questing for Quality Estimation A User Study

Cited by 9 publications

References 8 publications

Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output

Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output

The Role of Machine Translation Quality Estimation in the Post-Editing Workflow

Estrategia multidimensional para la selección de candidatos de traducción automática para posedición

Contact Info

Product

Resources

About