Fitting Sentence Level Translation Evaluation with Many Dense Features

Abstract

Sentence level evaluation in MT has turned out far more difficult than corpus level evaluation. Existing sentence level metrics employ a limited set of features, most of which are rather sparse at the sentence level, and their intricate models are rarely trained for ranking. This paper presents a simple linear model exploiting 33 relatively dense features, some of which are novel while others are known but seldom used, and train it under the learning-to-rank framework. We evaluate our metric on the standard WMT12 data showing that it outperforms the strong baseline METEOR. We also analyze the contribution of individual features and the choice of training data, language-pair vs. target-language data, providing new insights into this task.

chrF: character n-gram F-score for automatic MT evaluation

“…Recent investigations have shown that character level n-grams play an important role for automatic evaluation as a part of more complex metrics such as MTERATER (Parton et al, 2011) and BEER (Stanojević and Sima'an, 2014a;Stanojević and Sima'an, 2014b). However, they have not been investigated as an individual metric so far.…”

Evaluating the morphological competence of Machine Translation Systems

“…Finally, we have a set of systems that are optimized in order to improve target morphology. The automatic scores of the systems submitted at WMT'17 8 are in Table 4 where we report BLEU, BEER (Stanojević and Sima'an, 2014) and CharacTER (Wang et al, 2016). 9 We also computed a morphology accuracy for these systems.…”

Learning Morphological Normalization for Translation from and into Morphologically Rich Languages

“…BLEU scores for English-Czech Results are in Table 5, where we provide, in addition to BLEU, scores computed by BEER (Stanojević and Sima'an, 2014) and CharacTER (Wang et al, 2016). These two metrics proved to be more adapted to MRLs by Bojar et al (2016).…”