Improved English to Hindi Multimodal Neural Machine Translation

Abstract

Machine translation performs automatic translation from one natural language to another. Neural machine translation attains a state-ofthe-art approach in machine translation, but it requires adequate training data, which is a severe problem for low-resource language pairs translation. The concept of multimodal is introduced in neural machine translation (NMT) by merging textual features with visual features to improve low-resource pair translation. WAT2021 (Workshop on Asian Translation 2021) organizes a shared task of multimodal translation for English to Hindi. We have participated the same with team name CNLP-NITS-PP in two submissions: multimodal and text-only translation. This work investigates phrase pairs injection via data augmentation approach and attains improvement over our previous work at WAT2020 on the same task in both text-only and multimodal translation. We have achieved second rank on the challenge test set for English to Hindi multimodal translation where Bilingual Evaluation Understudy (BLEU) score of 39.28, Rank-based Intuitive Bilingual Evaluation Score (RIBES) 0.792097, and Adequacy-Fluency Metrics (AMFM) score 0.830230 respectively.

Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

“…is score was later improved again in the WAT2020 [12] task, where a BLEU score of 33.57 was obtained on the challenge test set. In [7], a bidirectional RNN of encoder type (BRNN) and a double-attention RNN of decoder type with default settings are used, and pretrained words are used to embed the data of monolingual corpus and additional parallel corpus IITB. In the WAT2021 multimodal translation task, [13] attempted to employ phrase pairs to improve English to Hindi translation effectiveness and performance.…”

“…In our experiments, we extract image features by different models and then add image features information to the encoder side of the baseline, fuse it with the initial word encoding information and position embedding information, and train an encoder containing image features, followed by a standard decoder module at the same time, to form a complete encoder and decoder model, thus improving the quality of English to Hindi translation. Comparing the experimental results, our approach improves 3 BLEU on the test set compared to the [7] method, and we also use different models to extract the image feature information and compare the impact of the extracted features on the translation results under different models.…”

Experiences of Adapting Multimodal Machine Translation Techniques for Hindi

“…Subsequently, it was observed that using multimodal methods achieves better results [42,61] compared to UNMT or image captioning. Doubly Attentive decoder [8] architecture, which was proven to generate a better translation for other languages, was adopted for English-Hindi [37,38]. They use pre-trained word embedding learned from IITB Corpus [34] to initialize the models and augment datasets by using Giza++ Tool [48], to obtain phrase pairs.…”

Neural machine translation for Sanskrit to Malayalam using morphology and evolutionary word sense disambiguation

“…High-quality translation involves a detailed concern of the source text and it enables effective and excellent language skills. Neural machine translation (NMT) is a new machine translation methodology [3]- [5] that helps to gain significantly in the evaluation of humans compared to statistical machine translation (SMT) systems. It's a whole end-to-end system designing one (or more) layers of machine translation into one huge neural network.…”