A Mixed Learning Objective for Neural Machine Translation

Abstract

Evaluation discrepancy and overcorrection phenomenon are two common problems in neural machine translation (NMT). NMT models are generally trained with word-level learning objective, but evaluated by sentence-level metrics. Moreover, the cross-entropy loss function discourages model to generate synonymous predictions and overcorrect them to ground truth words. To address these two drawbacks, we adopt multi-task learning and propose a mixed learning objective (MLO) which combines the strength of word-level and sentence-level evaluation without modifying model structure. At word-level, it calculates semantic similarity between predicted and ground truth words. At sentence-level, it computes probabilistic n-gram matching scores of generated translations. We also combine a loss-sensitive scheduled sampling decoding strategy with MLO to explore its extensibility. Experimental results on IWSLT 2016 German-English and WMT 2019 English-Chinese datasets demonstrate that our methodology can significantly promote translation quality. The ablation study shows that both word-level and sentence-level learning objective can improve BLEU scores. Furthermore, MLO is consistent with state-of-theart scheduled sampling methods and can achieve further promotion.

Neural Program Repair with Execution-based Backpropagation

“…Despite the difference in their inputs and neural models, those approaches are similar in the sense that they are all based on a typical NMT model formulated as an encoder-decoderattention architecture optimized with cross-entropy [11], [15], [16]. As a matter of fact, all the prior works on program repair are based on the NMT architecture with a cross-entropy loss [4]- [9].…”

“…The cross-entropy loss (a.k.a log loss) is a measure from information theory, building upon entropy and calculating the difference between two probability distributions. In sequence generation, the cross-entropy loss calculates the difference between the generated tokens and the human-written patch tokens in a strict pairwise matching manner [11], [16], [17]. In program repair patches, a low cross-entropy value means that the generated patch is syntactically close to the ground truth patch at the token level.…”

“…As discussed in subsection II-B and shown in Listing 1, because of this, there exists a discrepancy between training objectives and evaluation loss metric. In the field of neural machine translation (NMT), this problem is known as the overcorrection phenomenon of cross-entropy loss [11], [16]: cross-entropy based model tends to learn strictly identical translations and to overcorrect synonymous words which would be acceptable. In both cases, program repair and natural language translation, the problem is that the NMT-based model does not learn semantic relations.…”

“…The goal of semantic training is to let the patch generator be aware of program-specific knowledge (compilation and execution) beyond the syntactic loss computation at the tokenlevel. For that, we propose a mixed learning objective [16], where "mixed" means that the core learning objective is combined with two or more sub-learning objectives. In this paper, our mixed learning objective combines the core crossentropy objective with compilation and execution information.…”

“…In neural machine translation, Zhang et al [11] show the limitation of considering crossentropy loss and its tendency to overcorrect synonymous words and phrases. To relieve the problem, further research [11], [16] proposed to combine cross-entropy loss and add translation evaluation at the sentence level. In object detection, Ryou et al [25] proposed AnchorLoss to dynamically rescale the cross-entropy based on prediction difficulty.…”