On Vision Features in Multimodal Machine Translation

Abstract

Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models. In this work, we investigate the impact of vision models on MMT. Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and enhanced features (such as object-detection and image captioning). We develop a selective attention model to study the patch-level contribution of an image in MMT. On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the visual modality. Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased. Our code could be found at https: //github.com/libeineu/fairseq_mmt.

Linguistically Informed Post-processing for ASR Error correction in Sanskrit

“…Gated fusion techniques are widely used in combining the representations from different modalities as is done in some of the previous works [9]. In this method, for any input sample which consists of image I, source text S and target text T , the image features are obtained using the OpenAI CLIP's Vision Transformer(ViT) model [14] as ViT(I) and the textual embeddings are obtained from the standard transformer encoder as H S .…”

“…This method was implemented based on Multimodal Machine Translation where the sigmoid gate function was replaced 3: Sample Analysis with tanh. All the parameters were kept constant as in [9], except for the learning rate which was changed to 0.001 and the max updates to 800,000. For evaluation, the average of last 10 checkpoints was used for more reliable results.…”

“…Error correction can be viewed as similar to machine translation (MT) task where a sequence-to-sequence model like transformers [8] with large amounts of high-quality data can lead to excellent results. Inspired by multi-modal MT and other related tasks like visual text correction [9,10], in this paper, we propose a multi-modal ASR error correction method which utilizes visual information.…”

“…And the ASR transcripts were obtained from the Huggingface wav2vec model and the Google ASR API to show that the method is independent of the ASR model; (2)We propose two ways to utilize the visual information for ASR text correction. Firstly, a gated fusion method where the image features are concatenated with the textual embeddings, similar to previous works [9]. Secondly, we propose a prompt-based method to better utilize large-scale text data, where the captions from the images are used as prompts for ASR correction to provide more context.…”

Multimodal Chain-of-Thought Reasoning in Language Models

“…Then, we apply the gated fusion mechanism (Zhang et al, 2020;Wu et al, 2021;Li et al, 2022a) to fuse H language and H vision . The fused output H fuse ∈ R n×d is obtained by:…”