Semantic Correlation Mining between Images and Texts with Global Semantics and Local Mapping

Jiao, Xue; Du, Youtian; Shui, Hanbing

doi:10.1007/978-3-319-14442-9_48

Cited by 4 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Taking this one step further, Vempala and Preotiuc-Pietro (2019) take inspiration from Marsh and White (2003) and have studied image-text tweets for modeling relations that capture which modality adds information (or not) to the other. There is also work on estimating the semantic correlation (Zhang et al, 2008;Xue et al, 2015) and concept level relatedness between modalities (Yanai and Barnard, 2005) (e.g., between local image regions and words), but researchers have only recently (Chinnappa et al, 2019) started combining inter-modal relationships with computational modeling to investigate relations at a deeper level. Zhang et al (2018) focus on investigating non-literal relations between visual and textual persuasion for the automatic analysis of advertisements.…”

Section: Computational Sciencementioning

confidence: 99%

Understanding image-text relations and news values for multimodal news analysis

Cheema

Hakimov

Müller-Budack

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

The analysis of news dissemination is of utmost importance since the credibility of information and the identification of disinformation and misinformation affect society as a whole. Given the large amounts of news data published daily on the Web, the empirical analysis of news with regard to research questions and the detection of problematic news content on the Web require computational methods that work at scale. Today's online news are typically disseminated in a multimodal form, including various presentation modalities such as text, image, audio, and video. Recent developments in multimodal machine learning now make it possible to capture basic “descriptive” relations between modalities–such as correspondences between words and phrases, on the one hand, and corresponding visual depictions of the verbally expressed information on the other. Although such advances have enabled tremendous progress in tasks like image captioning, text-to-image generation and visual question answering, in domains such as news dissemination, there is a need to go further. In this paper, we introduce a novel framework for the computational analysis of multimodal news. We motivate a set of more complex image-text relations as well as multimodal news values based on real examples of news reports and consider their realization by computational approaches. To this end, we provide (a) an overview of existing literature from semiotics where detailed proposals have been made for taxonomies covering diverse image-text relations generalisable to any domain; (b) an overview of computational work that derives models of image-text relations from data; and (c) an overview of a particular class of news-centric attributes developed in journalism studies called news values. The result is a novel framework for multimodal news analysis that closes existing gaps in previous work while maintaining and combining the strengths of those accounts. We assess and discuss the elements of the framework with real-world examples and use cases, setting out research directions at the intersection of multimodal learning, multimodal analytics and computational social sciences that can benefit from our approach.

show abstract

Section: Computational Sciencementioning

confidence: 99%

Understanding image-text relations and news values for multimodal news analysis

Cheema

Hakimov

Müller-Budack

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…There are also some attempts to model semantic correlation between images and texts [23,26,27]. Xue et al [23] propose an approach to estimate semantic correlation by aligning the semantics of visual and textual blobs (local image regions and words) . In order to assign blobs to a document of another modality, they have to make the assumption that co-occurring image-text pairs do express the same semantics.…”

Section: Related Workmentioning

confidence: 99%

Estimating the Information Gap between Textual and Visual Representations

Henning

Ewerth

2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Photos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific publications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general illustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be described and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe cross-modal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach. ABSTRACTPhotos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific publications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general illustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be described and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe crossmodal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Xue et al [23] propose an approach to estimate semantic correlation by aligning the semantics of visual and textual blobs (local image regions and words) . In order to assign blobs to a document of another modality, they have to make the assumption that co-occurring image-text pairs do express the same semantics.…”

Section: Related Workmentioning

confidence: 99%

“…However, the focus of these approaches is to generate precise descriptions of the scene content that is depicted in an image, whereas they do not aim at providing complementary information. Some approaches investigate the interrelations of text and images, but they simply assume that an image is always semantically related to its surrounding text [23,27]. For the latter, it is easy to find examples showing that this is clearly not true, for example, by simply exploring today's online news in the World Wide Web.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Estimating the information gap between textual and visual representations

Henning

Ewerth

2017

Int J Multimed Info Retr

View full text Add to dashboard Cite

Photos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific publications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general illustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be described and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe cross-modal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general illustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be described and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe crossmodal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach.

show abstract

Semantic Correlation Mining between Images and Texts with Global Semantics and Local Mapping

Cited by 4 publications

References 12 publications

Understanding image-text relations and news values for multimodal news analysis

Understanding image-text relations and news values for multimodal news analysis

Estimating the Information Gap between Textual and Visual Representations

Estimating the information gap between textual and visual representations

Contact Info

Product

Resources

About