Machine Translation Evaluation Resources and Methods: A Survey

Abstract

We introduce the Machine Translation (MT) evaluation survey that contains both manual and automatic evaluation methods. The traditional human evaluation criteria mainly include the intelligibility, fidelity, fluency, adequacy, comprehension, and informativeness. The advanced human assessments include task-oriented measures, post-editing, segment ranking, and extended criteriea, etc. We classify the automatic evaluation methods into two categories, including lexical similarity scenario and linguistic features application. The lexical similarity methods contain edit distance, precision, recall, F-measure, and word order. The linguistic features can be divided into syntactic features and semantic features respectively. The syntactic features include part of speech tag, phrase types and sentence structures, and the semantic features include named entity, synonyms, textual entailment, paraphrase, semantic roles, and language models. The deep learning models for evaluation are very newly proposed. Subsequently, we also introduce the evaluation methods for MT evaluation including different correlation scores, and the recent quality estimation (QE) tasks for MT. This paper differs from the existing works (Dorr et al., 2009; EuroMatrix, 2007) from several aspects, by introducing some recent development of MT evaluation measures, the different classifications from manual to automatic evaluation measures, the introduction of recent QE tasks of MT, and the concise construction of the content.We hope this work will be helpful for MT researchers to easily pick up some metrics that are best suitable for their specific MT model development, and help MT evaluation researchers to get a general clue of how MT evaluation research developed. Furthermore, hopefully, this work can also shine some light on other evaluation tasks, except for translation, of NLP fields. 1

Can We Generate Shellcodes via Natural Language? An Empirical Study

“…In particular, this metric measures the degree of n-gram overlapping between the strings of words produced by the model and the human translation references at the corpus level. BLEU measures translation quality by the accuracy of translating n-grams to n-grams, for n-gram of size 1 to 4 [33]. The Exact match accuracy (ACC) is another automatic metric often used for evaluating neural machine translation [56,89,90,91].…”

Evaluating Commit Message Generation: To BLEU Or Not To BLEU?

“…Log-MNEXT = F-score * (1-Frag Penalty) (6) To get the F-Score, Precision (P) and recall (R) are calculated by assigning weights to the various unigram matches. = pred and = ref (7) where for a matcher type i ∈ {exact match, stemmed match, synonym match}, is the weight and is the number of matched unigrams. pred is the length of predicted text and ref is the length of reference text.…”

“…There are works comparing [3,7,14,18] automated evaluation metrics for machine translation. Our work is different from them because we discuss the suitability of metrics used in evaluating commit message generation tools.…”

EVIL: Exploiting Software via Natural Language

“…In particular, this metric measures the degree of n-gram overlapping between the strings of words produced by the model and the human translation references at the corpus level. BLEU measures translation quality by the accuracy of translating n-grams to n-grams, for n-gram of size 1 to 4 [57]. The Exact match accuracy (ACC) is another automatic metric often used for evaluating neural machine translation [7], [8], [15], [16].…”