Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Abstract

We introduce a Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation. Our decoder learns to attend to source-language words and parts of an image independently by means of two separate attention mechanisms as it generates words in the target language. We find that our model can efficiently exploit not just back-translated in-domain multi-modal data but also large general-domain text-only MT corpora. We also report state-of-the-art results on the Multi30k data set.

Is Neural Machine Translation the New State of the Art?

“…First, Section 3.1 looks at NMT for e-commerce, describing important parts of a more extended study that is reported in detail in Calixto et al (2017b). The second use case (Section 3.2) is an evaluation performed by Iconic Translation Machines Ltd.…”

“…MT Systems -Three different systems were compared in this experiment (1) a PBSMT baseline model built with the Moses SMT Toolkit (Koehn et al, 2007), (2) a text-only NMT model (NMT t ), and (3) a multi-modal NMT model (NMT m ), described in more detail in Calixto et al (2017b), which expands upon the text-only attentionbased model and introduces a visual component to incorporate local visual features.…”

DCU System Report on the WMT 2017 Multi-modal Machine Translation Task

“…By using an ensemble of four different multimodal NMT models trained on the translated Multi30k training data, we were able to obtain translations comparable to or even better than those obtained with the strong multi-modal NMT model of Calixto et al (2017a), which is pretrained on large amounts of WMT data and uses local image features.…”

“…Table 3: Results for the best model of Calixto et al (2017a), which is pre-trained on the English-German WMT 2015 (Bojar et al, 2015), and different combinations of multi-modal models, all trained on the original M30k T training data only, evaluated on the M30k T 2016 test set.…”

“…We ensembled different models by starting with one of Calixto et al (2017b)'s best performing multi-modal models on this data set, IMG D , and by adding new models to the ensemble one by one, until we reach a maximum of four independent models, all of which are trained separately and on the original M30k T training data only. We also report results for the best model of Calixto et al (2017a), which is pre-trained on the English-German WMT 2015 (Bojar et al, 2015) and uses local visual features extracted with the ResNet-50 network (He et al, 2015). We first note that adding more models to the ensemble seems to always improve translations by a large margin (∼ 3 BLEU/METEOR points).…”