Confidence Modeling for Neural Machine Translation

Aida, Taichi; Yamamoto, Kazumasa

doi:10.1109/ialp48816.2019.9037709

Cited by 1 publication

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve these problems, we have proposed a method to predict a quality of translated sentences by using NMT index as the confidence of the model in past research. 1 For the quality of the translated sentences, we employ the BLEU 2 score calculated from the number of word n-gram matches between the translated sentence and the reference translation. As a result of the past experiment, the loglikelihood normalized by the number of words showed a weak correlation with the BLEU score.…”

Section: Introductionmentioning

confidence: 99%

Estimating Machine Translation Quality of Any Input Sentence

Aida

Yamamoto

2020

Int. J. As. Lang. Proc.

Self Cite

View full text Add to dashboard Cite

Current methods of neural machine translation may generate sentences with different levels of quality. Methods for automatically evaluating translation output from machine translation can be broadly classified into two types: a method that uses human post-edited translations for training an evaluation model, and a method that uses a reference translation that is the correct answer during evaluation. On the one hand, it is difficult to prepare post-edited translations because it is necessary to tag each word in comparison with the original translated sentences. On the other hand, users who actually employ the machine translation system do not have a correct reference translation. Therefore, we propose a method that trains the evaluation model without using human post-edited sentences and in the test set, estimates the quality of output sentences without using reference translations. We define some indices and predict the quality of translations with a regression model. For the quality of the translated sentences, we employ the BLEU score calculated from the number of word [Formula: see text]-gram matches between the translated sentence and the reference translation. After that, we compute the correlation between quality scores predicted by our method and BLEU actually computed from references. According to the experimental results, the correlation with BLEU is the highest when XGBoost uses all the indices. Moreover, looking at each index, we find that the sentence log-likelihood and the model uncertainty, which are based on the joint probability of generating the translated sentence, are important in BLEU estimation.

show abstract

Section: Introductionmentioning

confidence: 99%