Attention-based Multimodal Neural Machine Translation

Abstract

Building models that take advantage of the hierarchical structure of language without a priori annotation is a longstanding goal in natural language processing. We introduce such a model for the task of machine translation, pairing a recurrent neural network grammar encoder with a novel atten-tional RNNG decoder and applying policy gradient reinforcement learning to induce unsupervised tree structures on both the source and target. When trained on character-level datasets with no explicit segmentation or parse annotation, the model learns a plausible segmentation and shallow parse, obtaining performance close to an attentional baseline.

Generating Image Descriptions using Multilingual Data

“…Supplementing these translation with information from the image provided only marginal improvements. For instance Huang et al (2016) re-ranked the translation output using image features and failed to achieve a higher METEOR score than the baseline. Similarly, systems developed for the WMT 2016 crosslingual image description multimodal task had access to one or more reference English descriptions of the image (in addition to the image itself) when attempting to generate a German caption, allowing them to use attention-based models that took advantage of both pieces of information.…”

Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation

“…Hitschler et al (2016) use image information by pivoting it on an external image captioning corpora. Most systems that make use of NMT add the image feature information into either the NMT encoder or decoder (Huang et al, 2016;Hokamp and Calixto, 2016), similar to Elliott et al (2015) with various enhancements. Marginal improvements according to automatic evaluation metrics were found only for approaches using re-ranking.…”

Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation

“…Multimodal NMT systems have been introduced (Elliott et al, 2015;Caglayan et al, 2016;Calixto et al, 2016;Huang et al, 2016) to incorporate visual information into NMT approaches, most of which condition the NMT on an image representation (typi-*P. Madhyastha and J. Wang contributed equally to this work.…”

“…Recent approaches to Multimodal NMT have used low level image features, including dense fully connected vectors and spatial convolutional representations from an image classification network (Elliott et al, 2015;Huang et al, 2016). They also incorporate attention mechanisms (Calixto et al, 2016).…”