The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional

Gwinnup, Jeremy; Sandvick, Joshua; Hutt, Michael; Erdmann, Grant; Duselis, John; Davis, James W.

doi:10.18653/v1/w18-6440

“…The CUNI submissions use two architectures based on the self-attentive Transformer model (Vaswani et al, 2017). For German and Czech, a language model is used to extract pseudo-in-ID Participating team AFRL-OHIOSTATE Air Force Research Laboratory & Ohio State University (Gwinnup et al, 2018) CUNI Univerzita Karlova v Praze (Helcl et al, 2018) LIUMCVC Laboratoire d'Informatique de l'Université du Maine & Universitat Autonoma de Barcelona Computer Vision Center (Caglayan et al, 2018) MeMAD Aalto University, Helsinki University & EURECOM (Grönroos et al, 2018) OSU-BAIDU Oregon State University & Baidu Research (Zheng et al, 2018) SHEF University of Sheffield UMONS Université de Mons (Delbrouck and Dupont, 2018) Table 5: Participants in the WMT18 multimodal machine translation shared task.…”

Section: Cuni (Task 1)mentioning

confidence: 99%

Findings of the Third Shared Task on Multimodal Machine Translation

Barrault¹,

Bougares²,

Specia³

et al. 2018

Proceedings of the Third Conference on Machine Translation: Shared Task Papers

View full text Add to dashboard Cite

We present the results from the third shared task on multimodal machine translation. In this task a source sentence in English is supplemented by an image and participating systems are required to generate a translation for such a sentence into German, French or Czech. The image can be used in addition to (or instead of) the source sentence. This year the task was extended with a third target language (Czech) and a new test set. In addition, a variant of this task was introduced with its own test set where the source sentence is given in multiple languages: English, French and German, and participating systems are required to generate a translation in Czech. Seven teams submitted 45 different systems to the two variants of the task. Compared to last year, the performance of the multimodal submissions improved, but text-only systems remain competitive.

show abstract

“…An n-best list from their SMT is reranked using a bi-directional NMT trained on the aforementioned source/target word sequences. Finally, Duselis et al (2017) and Gwinnup et al (2018) propose a pure retrieval system without any reranking involved. For a given image, they first obtain a set of candidate captions from a pretrained image captioning system.…”

Section: Reranking and Retrieval Based Approachesmentioning

confidence: 99%

Multimodal machine translation through visuals and speech

Sulubacak

¹

,

Çağlayan

²

,

Grönroos

³

et al. 2020

Machine Translation

View full text Add to dashboard Cite

Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data. The most prominent tasks in this area are spoken language translation, image-guided translation, and video-guided translation, which exploit audio and visual modalities, respectively. These tasks are distinguished from their monolingual counterparts of speech recognition, image captioning, and video captioning by the requirement of models to generate outputs in a different language. This survey reviews the major data resources for these tasks, the evaluation campaigns concentrated around them, the state of the art in end-to-end and pipeline approaches, and also the challenges in performance evaluation. The paper concludes with a discussion of directions for future research in these areas: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

show abstract

“…An n-best list from their SMT is reranked using a bidirectional NMT trained on the aforementioned source/target word sequences. Finally, Duselis et al (2017) and Gwinnup et al (2018) propose a pure retrieval system without any reranking involved. For a given image, they first obtain a set of candidate captions from a pretrained image captioning system.…”

Section: Reranking and Retrieval Based Approachesmentioning

confidence: 99%

Multimodal Machine Translation through Visuals and Speech

Sulubacak¹,

Çağlayan²,

Grönroos³

et al. 2019

Preprint

View full text Add to dashboard Cite

Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data. The most prominent tasks in this area are spoken language translation, image-guided translation, and video-guided translation, which exploit audio and visual modalities, respectively. These tasks are distinguished from their monolingual counterparts of speech recognition, image captioning, and video captioning by the requirement of models to generate outputs in a different language. This survey reviews the major data resources for these tasks, the evaluation campaigns concentrated around them, the state of the art in end-to-end and pipeline approaches, and also the challenges in performance evaluation. The paper concludes with a discussion of directions for future research in these areas: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

show abstract

The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional

Abstract: AFRL-Ohio State extends its usage of visual domain-driven machine translation for use as a peer with traditional machine translation systems. As a peer, it is enveloped into a system combination of neural and statistical MT systems to present a composite translation.

Cited by 4 publications

References 11 publications

Findings of the Third Shared Task on Multimodal Machine Translation

Findings of the Third Shared Task on Multimodal Machine Translation

Multimodal machine translation through visuals and speech

Multimodal Machine Translation through Visuals and Speech

Contact Info

Product

Resources

About