CUNI System for the WMT17 Multimodal Translation Task

Helcl, Jindřich; Libovický, Jindřich

doi:10.18653/v1/w17-4749

Cited by 16 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our proposed model, even though it is textual, produced competitive results with other multimodal models. The mixture-of-experts model outperformed several multimodal models, including another WMT submission [29]- [32]. Even in the out-of-domain dataset of COCO 2017, the mixture-of-experts model also performed reasonably well with a 28.0 BLEU score.…”

Section: Model Specification and Implementation Detailsmentioning

confidence: 90%

Leveraging Neural Caption Translation with Visually Grounded Paraphrase Augmentation

Effendi

Sakti

Sudoh

et al. 2020

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Since a concept can be represented by different vocabularies, styles, and levels of detail, a translation task resembles a many-to-many mapping task from a distribution of sentences in the source language into a distribution of sentences in the target language. This viewpoint, however, is not fully implemented in current neural machine translation (NMT), which is one-to-one sentence mapping. In this study, we represent the distribution itself as multiple paraphrase sentences, which will enrich the model context understanding and trigger it to produce numerous hypotheses. We use a visually grounded paraphrase (VGP), which uses images as a constraint of the concept in paraphrasing, to guarantee that the created paraphrases are within the intended distribution. In this way, our method can also be considered as incorporating image information into NMT without using the image itself. We implement this idea by crowdsourcing a paraphrasing corpus that realizes VGP and construct neural paraphrasing that behaves as expert models in a NMT. Our experimental results reveal that our proposed VGP augmentation strategies showed improvement against a vanilla NMT baseline.

show abstract

Section: Model Specification and Implementation Detailsmentioning

confidence: 90%

Leveraging Neural Caption Translation with Visually Grounded Paraphrase Augmentation

Effendi

Sakti

Sudoh

et al. 2020

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…The results concentrate around constrained systems, which only allow the use of parallel Multi30k corpus during training. A few studies experiment with using external resources (Calixto et al 2017;Helcl and Libovický 2017;Elliott and Kádár 2017;Grönroos et al 2018) for pretraining the MT system and then fine-tuning it on Multi30k, or directly training the system on the combination of Multi30k and the external resource. Two such unconstrained systems are also reported.…”

Section: Reranking and Retrieval Based Approachesmentioning

confidence: 99%

“…Lastly, if we take a look at the human evaluation rankings conducted throughout the WMT shared tasks, we see that the top three ranks for English→German and English→French are occupied by two unconstrained ensembles (Grönroos et al 2018;Helcl et al 2018b), the MLT Reranking and the DeepGRU (Delbrouck and Dupont 2018) systems in 2018. In 2017, the multiplicative interaction (Caglayan et al 2017a), unimodal NMT reranking (Zhang et al 2017), unconstrained Imagination (Elliott and Kádár 2017), encoder enrichment (Calixto and Liu 2017a) and hierarchical attention (Helcl and Libovický 2017) were ranked as top three, again for both language pairs.…”

Section: Comparison Of Approachesmentioning

confidence: 99%

Multimodal machine translation through visuals and speech

Sulubacak

Çağlayan

Grönroos

et al. 2020

Machine Translation

View full text Add to dashboard Cite

Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data. The most prominent tasks in this area are spoken language translation, image-guided translation, and video-guided translation, which exploit audio and visual modalities, respectively. These tasks are distinguished from their monolingual counterparts of speech recognition, image captioning, and video captioning by the requirement of models to generate outputs in a different language. This survey reviews the major data resources for these tasks, the evaluation campaigns concentrated around them, the state of the art in end-to-end and pipeline approaches, and also the challenges in performance evaluation. The paper concludes with a discussion of directions for future research in these areas: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

show abstract

“…We compare our approach against two state-ofthe-art multimodal translation systems: Caglayan et al (2017) modulate the target language word embeddings by an element-wise multiplication with a learned transformation of the visual data; Helcl and Libovický (2017) use a double attention model that learns to selectively attend to a combination of the source language and the visual data. Table 4 shows the results of the translation experiment.…”

Section: Modelsmentioning

confidence: 99%

Cross-lingual Visual Verb Sense Disambiguation

Gella¹,

Elliott²,

Keller³

2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Recent work has shown that visual context improves cross-lingual sense disambiguation for nouns. We extend this line of work to the more challenging task of cross-lingual verb sense disambiguation, introducing the Multi-Sense dataset of 9,504 images annotated with English, German, and Spanish verbs. Each image in MultiSense is annotated with an English verb and its translation in German or Spanish. We show that cross-lingual verb sense disambiguation models benefit from visual context, compared to unimodal baselines. We also show that the verb sense predicted by our best disambiguation model can improve the results of a text-only machine translation system when used for a multimodal translation task.

show abstract

CUNI System for the WMT17 Multimodal Translation Task

Cited by 16 publications

References 16 publications

Leveraging Neural Caption Translation with Visually Grounded Paraphrase Augmentation

Leveraging Neural Caption Translation with Visually Grounded Paraphrase Augmentation

Multimodal machine translation through visuals and speech

Cross-lingual Visual Verb Sense Disambiguation

Contact Info

Product

Resources

About