Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

Yu, Tiezheng; Dai, Wujiao; Liu, Zihan; Fung, Pascale

doi:10.18653/v1/2021.emnlp-main.326

Cited by 39 publications

(23 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We leverage multi-modal learning to co-represent features from both image and text. Common multimodal features co-representation techniques (Zeng et al, 2021) include fusion mechanism (early at feature level (Su et al, 2020), or late at decision/scoring level (Poria et al, 2016)), tensor factorization (Zadeh et al, 2017;Mai et al, 2019) and complex attention mechanisms which can be further classified as dot-product attention (Yu et al, 2021), multi-head attention (Cao et al, 2021;Wu et al, 2021), hierarchical attention (Pramanick et al, 2021), attention on attention among others. These techniques have proven to be effective in encoding multi-modal features.…”

Section: Multi-modal Feature Representationmentioning

confidence: 99%

MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

Gu¹,

Castro²,

Tyson³

2022

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)

View full text Add to dashboard Cite

Memes have become quite common in day-today communications on social media platforms. They often appear to be amusing, evoking and attractive to audiences. However, some memes containing malicious content can be harmful to targeted groups. In this paper, we study misogynous meme detection, a shared task in SemEval 2022 -Multimedia Automatic Misogyny Identification (MAMI). The challenge of misogynous meme detection is to co-represent multi-modal features. To tackle with this challenge, we propose a Multi-modal Multi-task Variational Au-toEncoder (MMVAE) to learn an effective corepresentation of visual and textual features in the latent space. Our goal is to automatically determine if a meme contains misogynous information and then identify its fine-grained category. Our model achieves F 1 scores of 0.723 on the MAMI sub-task A and 0.634 on sub-task B. We carry out comprehensive experiments on our model's architecture and show that our approach significantly outperforms several strong uni-modal and multi-modal approaches. Our code is released on github 1 .

show abstract

Section: Multi-modal Feature Representationmentioning

confidence: 99%

MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

Gu¹,

Castro²,

Tyson³

2022

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)

View full text Add to dashboard Cite

show abstract

“…Besides these pre-trained models, various models using reinforcement learning [12] [13], topic model [14] [15], multimodal information [16], attention head masking [17], information theory [18], extraction-and-paraphrasing [19], entity aggregation [20], factuality consistency [21] [22] [23] [24] [25] [26], deep communicating agents [27], sentence correspondence [28], graph [29] [30] [31], and bottom-up approach [32] were proposed in abstractive summarization. Because these models are not directly related to our proposed model, we do not compared with them in this paper.…”

Section: Related Workmentioning

confidence: 99%

Learning Cluster Patterns for Abstractive Summarization

Jo¹,

Kim²,

On³

2022

Preprint

View full text Add to dashboard Cite

Nowadays, pre-trained sequence-to-sequence models such as BERTSUM and BART have shown state-of-the-art results in abstractive summarization. In these models, during fine-tuning, the encoder transforms sentences to context vectors in the latent space and the decoder learns the summary generation task based on the context vectors. In our approach, we consider two clusters of salient and non-salient context vectors, using which the decoder can attend more to salient context vectors for summary generation. For this, we propose a novel clustering transformer layer between the encoder and the decoder, which first generates two clusters of salient and non-salient vectors, and then normalizes and shrinks the clusters to make them apart in the latent space. Our experimental result shows that the proposed model outperforms the existing BART model by learning these distinct cluster patterns, improving up to 4% in ROUGE and 0.3% in BERTScore on average in CNN/DailyMail and XSUM data sets.

show abstract

“…Additionally, as presented in Table 5, we also run VLKD on the abstractive summarization task to evaluate its NLG performance, since BART-based methods excel on the summarization (Lewis et al, 2020;Dou et al, 2021;Yu et al, 2021b). The gap between VLKD and its backbone BART is negligible.…”

Section: Evaluation Of Nlu and Nlgmentioning

confidence: 99%

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Dai¹,

Hou²,

Shang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The recent large-scale vision-language pretraining (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data-and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7× fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

show abstract

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

Cited by 39 publications

References 39 publications

MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

Learning Cluster Patterns for Abstractive Summarization

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Contact Info

Product

Resources

About