A Survey on Multi-modal Summarization

Abstract

The new era of technology has brought us to the point where it is convenient for people to share their opinions over an abundance of platforms. These platforms have a provision for the users to express themselves in multiple forms of representations, including text, images, videos, and audio. This, however, makes it difficult for users to obtain all the key information about a topic, making the task of automatic multi-modal summarization (MMS) essential. In this paper, we present a comprehensive survey of the existing research in the area of MMS, covering various modalities like text, image, audio, and video. Apart from highlighting the different evaluation metrics and datasets used for the MMS task, our work also discusses the current challenges and future directions in this field.

Evaluating MediBetter: A Mobile Application for Health Monitoring and Medication Management

“…To evaluate the quality of generated summaries generated by both the user and ChatGPT, we employed extensive human assessments. We adopted standardized metrics commonly used in previous studies [5,6,7,8] . These metrics encompass the following dimensions: (1) Informativeness or Relevance: Quantifying the ability of the summary to retain important and relevant facts and information.…”

“…(5) Consistency or Factuality: Verifying the factual accuracy of the summary in comparison to the source article. (6) Contradiction: Identifying instances where information within the summaries contradicts other information or disagrees with another piece of information. This metric measures the summary should contain few contradiction, with a higher score indicating a lower level of contradiction.…”

Generative AI models in time varying biomedical data: a systematic review (Preprint)

“…Both traditional NLP and foundation models can be used to perform clinical NLP tasks such as clinical concept extraction (or named entity recognition NER), medical relation extraction (MRE), semantic textual similarity (STS), natural language inference (NLI), medical question answering (MQA) and Medical Report Summarization. The domain of medical report summarization is a popular area of research with numerous papers published on the use of NLP and foundation models to summarize medical reports, healthcare records and medical dialogues [97]. A radiology report is a medical document that contains the details of an imaging study (such as X-ray, MRI, etc).…”

Multimodal Abstractive Summarization using bidirectional encoder representations from transformers with attention mechanism

“…images and videos which is coming from various domains like science, blogs, news, finance, medicine, academics etc. [ 1 , 2 ]. For example, Facebook, Instagram and YouTube has millions of users who posted billons of videos on daily basis.…”