The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach

Duselis, John; Hutt, Michael; Gwinnup, Jeremy; Davis, James W.; Sandvick, Joshua

doi:10.18653/v1/w17-4748

“…An n-best list from their SMT is reranked using a bi-directional NMT trained on the aforementioned source/target word sequences. Finally, Duselis et al (2017) and Gwinnup et al (2018) propose a pure retrieval system without any reranking involved. For a given image, they first obtain a set of candidate captions from a pretrained image captioning system.…”

Section: Reranking and Retrieval Based Approachesmentioning

confidence: 99%

Multimodal machine translation through visuals and speech

Sulubacak

¹

,

Çağlayan

²

,

Grönroos

³

et al. 2020

Machine Translation

View full text Add to dashboard Cite

Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data. The most prominent tasks in this area are spoken language translation, image-guided translation, and video-guided translation, which exploit audio and visual modalities, respectively. These tasks are distinguished from their monolingual counterparts of speech recognition, image captioning, and video captioning by the requirement of models to generate outputs in a different language. This survey reviews the major data resources for these tasks, the evaluation campaigns concentrated around them, the state of the art in end-to-end and pipeline approaches, and also the challenges in performance evaluation. The paper concludes with a discussion of directions for future research in these areas: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

show abstract

“…An n-best list from their SMT is reranked using a bidirectional NMT trained on the aforementioned source/target word sequences. Finally, Duselis et al (2017) and Gwinnup et al (2018) propose a pure retrieval system without any reranking involved. For a given image, they first obtain a set of candidate captions from a pretrained image captioning system.…”

Section: Reranking and Retrieval Based Approachesmentioning

confidence: 99%

Multimodal Machine Translation through Visuals and Speech

Sulubacak¹,

Çağlayan²,

Grönroos³

et al. 2019

Preprint

View full text Add to dashboard Cite

Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data. The most prominent tasks in this area are spoken language translation, image-guided translation, and video-guided translation, which exploit audio and visual modalities, respectively. These tasks are distinguished from their monolingual counterparts of speech recognition, image captioning, and video captioning by the requirement of models to generate outputs in a different language. This survey reviews the major data resources for these tasks, the evaluation campaigns concentrated around them, the state of the art in end-to-end and pipeline approaches, and also the challenges in performance evaluation. The paper concludes with a discussion of directions for future research in these areas: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

show abstract

“…Here is an overview of the VMT system submitted to the WMT17 submission (Duselis et al, 2017). This system architecture assumes a captionator can be trained in a target language to give meaningful output in the form of a set of the most probable n target language candidate captions.…”

Section: The Afrl-ohio State Wmt17 Submissionmentioning

confidence: 99%

“…Most of the submissions to the Second Conference on Machine Translation (WMT17) Multimodal submissions for Task 1 (Elliott et al, 2017) used the visual domain to enhance machine translation of the image+caption pair. The exception was a Visual Machine Translation (VMT) system where the image is the driver for the translation (Duselis et al, 2017). While the scores for this submission did not approach baseline, except by human scoring, it did introduce the concept that the visual domain can approach parity with the traditional text based MT systems.…”

Section: Introductionmentioning

confidence: 96%