Human versus automatic quality evaluation of NMT and PBSMT

Shterionov, Dimitar; Superbo, Riccardo; Nagle, Pat; Casanellas, Laura; O'Dowd, Tony; Way, Andy

doi:10.1007/s10590-018-9220-z

Cited by 34 publications

(22 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As I note in Way (2018), it simply cannot be the case that a 2-point improvement in BLEU scorealmost an irrelevance on a real industrial translation use-case which was typically seen in WMT-2016 where NMT systems swept the board on all tasks and language-pairs , can be reflective of the improvements in word order and lexical selection noted by Bentivogli et al (2016). Note that Shterionov et al (2018) actually computed the degree of underestimation in quality of three popular automatic evaluation metrics -BLEU, METEOR and TERshowing that for NMT, this may be up to 50%.…”

Section: Iii2 Does Mt Evaluation Need To Change With Nmt Coming Onstmentioning

confidence: 99%

Machine translation: Where are we at today?

Way¹

2020

The Bloomsbury Companion to Language Industry Studies

Self Cite

View full text Add to dashboard Cite

Section: Iii2 Does Mt Evaluation Need To Change With Nmt Coming Onstmentioning

confidence: 99%

Machine translation: Where are we at today?

Way¹

2020

The Bloomsbury Companion to Language Industry Studies

Self Cite

View full text Add to dashboard Cite

“…The former is typically achieved by comparing an MT output to a reference translation and generating metrics such as BLEU [7], whereas the latter concerns the human assessment of an MT output, such as the annotations of errors or the ranking of translations. Although manual assessment is much more time-consuming, its automatic variant has often been criticized as being problematic [8], having the tendency to underestimate the quality of NMT systems [9], or not even being a representation of the actual quality of the translation [1]. Sceptical of the reliability of automatic assessment, Matusov [1] performed a manual assessment of literary NMT with his own classification system, specifically designed for NMT.…”

Section: Related Researchmentioning

confidence: 99%

Gutenberg Goes Neural: Comparing Features of Dutch Human Translations with Raw Neural Machine Translation Outputs in a Corpus of English Literary Classics

et al. 2020

View full text Add to dashboard Cite

Due to the growing success of neural machine translation (NMT), many have started to question its applicability within the field of literary translation. In order to grasp the possibilities of NMT, we studied the output of the neural machine system of Google Translate (GNMT) and DeepL when applied to four classic novels translated from English into Dutch. The quality of the NMT systems is discussed by focusing on manual annotations, and we also employed various metrics in order to get an insight into lexical richness, local cohesion, syntactic, and stylistic difference. Firstly, we discovered that a large proportion of the translated sentences contained errors. We also observed a lower level of lexical richness and local cohesion in the NMTs compared to the human translations. In addition, NMTs are more likely to follow the syntactic structure of a source sentence, whereas human translations can differ. Lastly, the human translations deviate from the machine translations in style.

show abstract

“…Wu et al 2016;Junczys-Dowmunt et al 2016;Crego et al 2016;Castilho et al 2017). However, recent research suggests that automatic metrics may not always be suitable: Shterionov et al (2018) compared automatic evaluation scores to human evaluation and noticed that the automatic scores underestimated the quality of NMT systems. Other studies, like Castilho et al (2017), report mixed results using different automatic (HTER, BLEU) and human evaluation metrics (fluency, adequacy) -SMT systems outperformed NMT in two case studies out of three -and point out that results vary depending on domain and language pair.…”

Section: Nmt Output and Qualitymentioning

confidence: 99%

A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output

Koponen

Salmi

Nikulin

2019

Machine Translation

View full text Add to dashboard Cite

This paper presents a comparison of post-editing (PE) changes performed on English-to-Finnish neural (NMT), rule-based (RBMT) and statistical machine translation (SMT) output, combining a product-based and a process-based approach. A total of 33 translation students acted as participants in a PE experiment providing both post-edited texts and edit process data. Our product-based analysis of the postedited texts shows statistically significant differences in the distribution of edit types between machine translation systems. Deletions were the most common edit type for the RBMT, insertions for the SMT, and word form changes as well as word substitutions for the NMT system. The results also show significant differences in the correctness and necessity of the edits, particularly in the form of a large number of unnecessary edits in the RBMT output. Problems related to certain verb forms and ambiguity were observed for NMT and SMT, while RBMT was more likely to handle them correctly. Process-based comparison of effort indicators shows a slight increase of keystrokes per word for NMT output, and a slight decrease in average pause length for NMT compared to RBMT and SMT in specific text blocks. A statistically significant difference was observed in the number of visits per sub-segment, which is lower for NMT than for RBMT and SMT. The results suggest that although different types of edits were needed to outputs from NMT, RBMT and SMT systems, the difference is not necessarily reflected in process-based effort indicators.

show abstract

Human versus automatic quality evaluation of NMT and PBSMT

Cited by 34 publications

References 19 publications

Machine translation: Where are we at today?

Machine translation: Where are we at today?

Gutenberg Goes Neural: Comparing Features of Dutch Human Translations with Raw Neural Machine Translation Outputs in a Corpus of English Literary Classics

A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output

Contact Info

Product

Resources

About