DCU-UvA Multimodal MT System Report

Abstract

We present a doubly-attentive multimodal machine translation model. Our model learns to attend to source language and spatial-preserving CONV 5,4 visual features as separate attention mechanisms in a neural translation model. In image description translation experiments (Task 1), we find an improvement of 2.3 Meteor points compared to initialising the hidden state of the decoder with only the FC 7 features and 2.9 Meteor points compared to a text-only neural machine translation baseline, confirming the useful nature of attending to the CONV 5,4 features.

The role of image representations in vision to language tasks

“…Most existing work obtain the best results by combining the penultimate layer of the CNN (via concatenation, summation, etc.) with the final state of the source sentence representation and using it to initialize the target RNN (Caglayan et al, 2016;Calixto et al, 2016;Huang et al, 2016). Recent work also explores an attention mechanism where they use lower level CNN features of the images, such as a convolutional layer, and condition the source and the target sentences on the image features (Calixto et al, 2016;Calixto et al, 2017).…”

“…with the final state of the source sentence representation and using it to initialize the target RNN (Caglayan et al, 2016;Calixto et al, 2016;Huang et al, 2016). Recent work also explores an attention mechanism where they use lower level CNN features of the images, such as a convolutional layer, and condition the source and the target sentences on the image features (Calixto et al, 2016;Calixto et al, 2017). The intuition here is that the lower-level CNN features capture information about different areas of the images and an attention mechanism could learn to attend to specific regions while both encoding the source and decoding the target sentence.…”

Proceedings of the Sixth Workshop on Vision and Language

“…We then briefly discuss the doubly-attentive multi-modal NMT model we use in our experiments ( §2.3), which is comparable to the model evaluated by Calixto et al (2016) and further detailed and analysed in Calixto et al (2017a). …”

Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation

“…Multimodal NMT systems have been introduced (Elliott et al, 2015;Caglayan et al, 2016;Calixto et al, 2016;Huang et al, 2016) to incorporate visual information into NMT approaches, most of which condition the NMT on an image representation (typi-*P. Madhyastha and J. Wang contributed equally to this work.…”

“…They also incorporate attention mechanisms (Calixto et al, 2016). However, the effect of image features or the efficacy of the representational contribution is still an open research question.…”