Multimodal Machine Translation with Embedding Prediction

Hirasawa, Tosho; Yamagishi, Hayahide; Matsumura, Yutaka; Komachi, Mamoru

doi:10.18653/v1/n19-3012

Cited by 8 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…They also indicate that the improvement achieved using the pre-trained word embedding decreases as the training data increases. Hirasawa et al [15] introduced an MMT model with embedding prediction that provided substantial performance improvement. However, many studies have proven that the vectors in a pre-trained word embedding distribute unevenly in a narrow conical subspace rather than evenly in the entire space.…”

Section: A Word Embeddingmentioning

confidence: 99%

“…Their approach achieves not only faster training convergence without decreasing translation performance, but also more accurate translation of rare words. This technique has been extended to an MMT model by [15], with resulting improvement in translation quality.…”

Section: B Monolingual Corpora Augmented Nmtmentioning

confidence: 99%

“…Bi-text Image parallel data Text Image NMT [3,27] BERT-fused NMT [26] MMT MMT w/ back-translation [8,30] VTLM [11] MMT w/ embedding prediction ( [15], Section. III) LXMERT-fused MMT (Section.…”

Section: Multi30kmentioning

confidence: 99%

See 2 more Smart Citations

Pre-Trained Word Embedding and Language Model Improve Multimodal Machine Translation: A Case Study in Multi30K

et al. 2022

Self Cite

View full text Add to dashboard Cite

Multimodal machine translation (MMT) is an attractive application of neural machine translation (NMT) that is commonly incorporated with image information. However, the MMT models proposed thus far have only comparable or slightly better performance than their text-only counterparts. One potential cause of this infeasibility is a lack of large-scale data. Most previous studies mitigate this limitation by employing large-scale textual parallel corpora, which are more accessible than multimodal parallel corpora, in various ways. However, these corpora are still available on only a limited scale in low-resource language pairs or domains. In this study, we leveraged monolingual (or multimodal monolingual) corpora, which are available at scale in most languages and domains, to improve MMT models. Our approach follows that of previous unimodal works that use monolingual corpora to train the word embedding or language model and incorporate them into NMT systems. While these methods demonstrated the advantage of using pre-trained representations, there is still room for MMT models to improve. To this end, our system employs debiasing procedures for the word embedding and multimodal extension of the language model (visual-language model, VLM) to make better use of the pre-trained knowledge in the MMT task. The results of evaluations conducted on the de facto MMT dataset for the English-German translation indicate that the improvement obtained using well-tailored word embedding and VLM is approximately +1.84 BLEU and +1.63 BLEU, respectively. The evaluation on multiple language pairs reveals their adoptability across the languages. Beyond the success of our system, we also conducted an extensive analysis on VLM manipulation and showed promising areas for developing better MMT models by exploiting VLM; some benefits brought by either modality are missing, and MMT with VLM generates less fluent translations. Our code is available at https://github.com/toshohirasawa/mmt-with-monolingual-data.INDEX TERMS Multimodal machine translation, natural language processing, neural machine translation, I. INTRODUCTION

show abstract

Section: A Word Embeddingmentioning

confidence: 99%

Section: B Monolingual Corpora Augmented Nmtmentioning

confidence: 99%

See 1 more Smart Citation

Pre-Trained Word Embedding and Language Model Improve Multimodal Machine Translation: A Case Study in Multi30K

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…We also have witnessed two proposed categories for MMT from the perspective of cross-modal learning approaches, which either explicitly transform visual features and textual embeddings from one modality to the other at both training and inference (Caglayan et al 2017;Yin et al 2020), or implicitly align the visual and textual modalities to generate vision-aware textual features at training. Unlike the explicit approaches, the implicit cross-modal learning methods do not require images as input at inference, taking the image features as latent variables across different languages (Elliott and Kádár 2017;Calixto, Rios, and Aziz 2019;Hirasawa et al 2019), which also serves as a latent scheme for unsupervised MMT (Lee et al 2018). Despite of the success of plenty of models on Multi30K, an interesting finding is that the visual modality is not fully exploited and only marginally beneficial to machine translation (Caglayan et al 2017;Ive, Madhyastha, and Specia 2019).…”

Section: Mmt Without Text Maskingmentioning

confidence: 99%

Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding

Wang

Xiong

2021

Preprint

View full text Add to dashboard Cite

Visual context provides grounding information for multimodal machine translation (MMT). However, previous MMT models and probing studies on visual features suggest that visual information is less explored in MMT as it is often redundant to textual information. In this paper, we propose an object-level visual context modeling framework (OVC) to efficiently capture and explore visual information for multimodal machine translation. With detected objects, the proposed OVC encourages MMT to ground translation on desirable visual objects by masking irrelevant objects in the visual modality. We equip the proposed with an additional object-masking loss to achieve this goal. The object-masking loss is estimated according to the similarity between masked objects and the source texts so as to encourage masking source-irrelevant objects. Additionally, in order to generate vision-consistent target words, we further propose a visionweighted translation loss for OVC. Experiments on MMT datasets demonstrate that the proposed OVC model outperforms state-of-the-art MMT models and analyses show that masking irrelevant objects helps grounding in MMT.

show abstract

“…A monolingual corpus can be collected relatively easily, and has been known to contribute to improved statistical machine translation [2]. Various attempts to employ a monolingual corpus have involved the following: pre-training of a translation model [12], initialization of distributed word representation [4,11], and construction of a pseudo-parallel corpus by back-translation [14].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Fusion: Attentional Language Model for Neural Machine Translation

Kurosawa

Komachi

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Neural Machine Translation (NMT) can be used to generate fluent output. As such, language models have been investigated for incorporation with NMT. In prior investigations, two models have been used: a translation model and a language model. The translation model's predictions are weighted by the language model with a hand-crafted ratio in advance. However, these approaches fail to adopt the language model weighting with regard to the translation history. In another line of approach, language model prediction is incorporated into the translation model by jointly considering source and target information. However, this line of approach is limited because it largely ignores the adequacy of the translation output. Accordingly, this work employs two mechanisms, the translation model and the language model, with an attentive architecture to the language model as an auxiliary element of the translation model. Compared with previous work in English-Japanese machine translation using a language model, the experimental results obtained with the proposed Dynamic Fusion mechanism improve BLEU and Rank-based Intuitive Bilingual Evaluation Scores (RIBES) scores. Additionally, in the analyses of the attention and predictivity of the language model, the Dynamic Fusion mechanism allows predictive language modeling that conforms to the appropriate grammatical structure.

show abstract

Multimodal Machine Translation with Embedding Prediction

Cited by 8 publications

References 12 publications

Pre-Trained Word Embedding and Language Model Improve Multimodal Machine Translation: A Case Study in Multi30K

Pre-Trained Word Embedding and Language Model Improve Multimodal Machine Translation: A Case Study in Multi30K

Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding

Dynamic Fusion: Attentional Language Model for Neural Machine Translation

Contact Info

Product

Resources

About