Data-driven annotation of binary MT quality estimation corpora based on human post-editions

Turchi, Marco; Negri, Matteo; Federico, Marcello

doi:10.1007/s10590-014-9162-z

Cited by 12 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on their postedits, the raw MT output samples are then labeled as 'good' or 'bad' by considering the HTER (Snover et al, 2006) calculated between raw MT output and its post-edited version. 3 Our labeling criterion follows the empirical findings of (Turchi et al, 2013;Turchi et al, 2014b), which indicate an HTER value of 0.4 as boundary between posteditable (HTER ≤ 0.4) and useless suggestions (HTER> 0.4). Then, to model the subjective concept of quality of different subjects, for of each translator we train a separate binary QE classifier on the labeled samples.…”

Section: Getting Binary Quality Labelsmentioning

confidence: 98%

MT Quality Estimation for Computer-assisted Translation: Does it Really Help?

Turchi

Negri

Federico

2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

Self Cite

View full text Add to dashboard Cite

The usefulness of translation quality estimation (QE) to increase productivity in a computer-assisted translation (CAT) framework is a widely held assumption (Specia, 2011;Huang et al., 2014). So far, however, the validity of this assumption has not been yet demonstrated through sound evaluations in realistic settings. To this aim, we report on an evaluation involving professional translators operating with a CAT tool in controlled but natural conditions. Contrastive experiments are carried out by measuring post-editing time differences when: i) translation suggestions are presented together with binary quality estimates, and ii) the same suggestions are presented without quality indicators. Translators' productivity in the two conditions is analysed in a principled way, accounting for the main factors (e.g. differences in translators' behaviour, quality of the suggestions) that directly impact on time measurements. While the general assumption about the usefulness of QE is verified, significance testing results reveal that real productivity gains can be observed only under specific conditions.

show abstract

Section: Getting Binary Quality Labelsmentioning

confidence: 98%

MT Quality Estimation for Computer-assisted Translation: Does it Really Help?

Turchi

Negri

Federico

2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

Self Cite

View full text Add to dashboard Cite

show abstract

“…Three different tokens are used, namely "no post-edit" (no edits are required), "light post-edit" (minimal edits are required), and "heavy post-edit" (a large number of edits are required. At training time, the instances are labelled based on the TER computed between the MT output and its post-edited version, with the boundary between light and heavy post-edit set to TER=0.4 based on the findings reported in (Turchi et al, 2013;Turchi et al, 2014). At test time, tokens are predicted with two approaches.…”

Section: Participantsmentioning

confidence: 99%

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Chatterjee¹,

Federmann²,

Negri³

et al. 2019

Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)

Self Cite

View full text Add to dashboard Cite

We present the results from the 5 th round of the WMT task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a "black-box" machine translation system by learning from human corrections. Keeping the same general evaluation setting of the previous four rounds, this year we focused on two language pairs (English-German and English-Russian) and on domain-specific data (Information Technology). For both the language directions, MT outputs were produced by neural systems unknown to participants. Seven teams participated in the English-German task, with a total of 18 submitted runs. The evaluation, which was performed on the same test set used for the 2018 round, shows further progress in APE technology: 4 teams achieved better results than last year's winning system, with improvements up to-0.78 TER and +1.23 BLEU points over the baseline. Two teams participated in the English-Russian task submitting 2 runs each. On this new language direction, characterized by a higher quality of the original translations, the task proved to be particularly challenging. Indeed, none of the submitted runs improved the very high results of the strong system used to produce the initial translations (16.16 TER, 76.20 BLEU).

show abstract

“…For instance, based on the empirical findings reported in(Turchi et al, 2013(Turchi et al, , 2014, TER=0.4 is the threshold that, for human post-editors, separates the "post-editable" translations from those that require complete rewriting from scratch.…”

mentioning

confidence: 99%

Findings of the WMT 2018 Shared Task on Automatic Post-Editing

Chatterjee¹,

Negri²,

Rubino³

et al. 2018

Proceedings of the Third Conference on Machine Translation: Shared Task Papers

Self Cite

View full text Add to dashboard Cite

We present the results from the fourth round of the WMT shared task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a "black-box" machine translation system by learning from human corrections. Keeping the same general evaluation setting of the three previous rounds, this year we focused on one language pair (English-German) and on domainspecific data (Information Technology), with MT outputs produced by two different paradigms: phrase-based (PBSMT) and neural (NMT). Five teams submitted respectively 11 runs for the PBSMT subtask and 10 runs for the NMT subtask. In the former subtask, characterized by original translations of lower quality, top results achieved impressive improvements, up to-6.24 TER and +9.53 BLEU points over the baseline "do-nothing" system. The NMT subtask proved to be more challenging due to the higher quality of the original translations and the availability of less training data. In this case, top results show smaller improvements up to-0.38 TER and +0.8 BLEU points.

show abstract

Data-driven annotation of binary MT quality estimation corpora based on human post-editions

Cited by 12 publications

References 19 publications

MT Quality Estimation for Computer-assisted Translation: Does it Really Help?

MT Quality Estimation for Computer-assisted Translation: Does it Really Help?

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Findings of the WMT 2018 Shared Task on Automatic Post-Editing

Contact Info

Product

Resources

About