The Role of Machine Translation Quality Estimation in the Post-Editing Workflow

Abstract

As Machine Translation (MT) becomes increasingly ubiquitous, so does its use in professional translation workflows. However, its proliferation in the translation industry has brought about new challenges in the field of Post-Editing (PE). We are now faced with a need to find effective tools to assess the quality of MT systems to avoid underpayments and mistrust by professional translators. In this scenario, one promising field of study is MT Quality Estimation (MTQE), as this aims to determine the quality of an automatic translation and, indirectly, its degree of post-editing difficulty. However, its impact on the translation workflows and the translators’ cognitive load is still to be fully explored. We report on the results of an impact study engaging professional translators in PE tasks using MTQE. To assess the translators’ cognitive load we measure their productivity both in terms of time and effort (keystrokes) in three different scenarios: translating from scratch, post-editing without using MTQE, and post-editing using MTQE. Our results show that good MTQE information can improve post-editing efficiency and decrease the cognitive load on translators. This is especially true for cases with low MT quality.

The phraseology of ‘frontline’ in the Covid-19 pandemic

“…Grouping lexical items under these semantic types makes it easier to generalise the selectional preferences of words. CPA has predominantly been carried out on verbs for lexicographical purposes, as in the Pattern Dictionary of English Verbs (http://pdev.org.uk), but it has also been carried out, minimally and experimentally, on nouns [23], prepositions [24] and modifiers [25], and has been applied to a range of problems, including machine translation evaluation [26], semantic processing [27] and discourse analysis [28]. The theory of norms and exploitations (TNE) that underpins CPA proposes a "double helix" representation of meaning, with norms on one side and exploitations of those norms on the other.…”

Comparing MTQE Scores with Fuzzy Match Percentages from CAT Tools

“…It is worth mentioning that previous testing in user settings has shown contradicting results on the usefulness of displaying MTQE scores to post-editors [1,6,9,10], the interest for MTQE is still rising there have been suggestions for the fact that MT confidence score may be useful during translation and localization workflows and Moorkens et al report that most post-editors who took part in their survey would also like "to be presented with confidence scores for each target text segment from the MT engine." [3].…”

“…One popular such method of displaying the result consists in showing the MTQE analysis results to the post editors with a "traffic light" coding [1,6,10] representing a binary indication of the estimated quality of the MT raw output. The linguist in this case is given an indication such as:…”

“…-Good/green: the MT output can be used for post-editing according to the automatic estimation as it requires little to no changes; -Bad/red: the MT suggestion should be discarded as a human translation from scratch would require less time/effort. Béchara et al [1] based their traffic light system on fuzzy match scores as "translators are more used to working with TM leveraging and fuzzy matches" and used a 75% quality threshold to predict the good/bad evaluation. In their study, however, fuzzy match percentages are not shown to the post-editors who are then provided with the quality estimation by means of color codes.…”

Analysis of Feature Extraction Model for Machine Korean Translation Judging System

“…Productivity may be increased by reducing the time and expense involved in manual quality control procedures. It is crucial to keep in mind that the effectiveness of the machine translation judging system for Korea relies on how accurate and reliable the machine translation system is and how accurate the evaluation metrics are [10]. In the study [11], they created a lexical semantic network by hand for the unique features of Korean as a knowledge foundation for our Korean morphological analysis and word-sense disambiguation system.…”