Effective Multimodality Fusion Framework for Cross-Media Topic Detection

Chu, Lingyang; Zhang, Yanyan; Li, Guorong; Wang, Shuhui; Zhang, Weigang; Huang, Qingming

doi:10.1109/tcsvt.2014.2347551

Cited by 35 publications

(11 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on Markov random field, this model learns a set of shared topics across the modalities. Chu et al [12] developed a flexible multimodality graph (MMG) fusion framework to fuse the complex multi-modal data from different media and a topic recovery approach to effectively detect topics from cross-media data.…”

Section: Multi-modal and Cross-modal Retrievalmentioning

confidence: 99%

An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

et al. 2019

View full text Add to dashboard Cite

Due to the rapid development of mobile Internet techniques, cloud computation and popularity of online social networking and location-based services, massive amount of multimedia data with geographical information is generated and uploaded to the Internet. In this paper, we propose a novel type of cross-modal multimedia retrieval called geo-multimedia cross-modal retrieval which aims to search out a set of geo-multimedia objects based on geographical distance proximity and semantic similarity between different modalities. Previous studies for cross-modal retrieval and spatial keyword search cannot address this problem effectively because they do not consider multimedia data with geo-tags and do not focus on this type of query. In order to address this problem efficiently, we present the definition of kNN geo-multimedia cross-modal query at the first time and introduce relevant conceptions such as cross-modal semantic representation space. To bridge the semantic gap between different modalities, we propose a method named cross-modal semantic matching which contains two important component, i.e., CorrProj and LogsTran, which aims to construct a common semantic representation space for cross-modal semantic similarity measurement. Besides, we designed a framework based on deep learning techniques to implement common semantic

show abstract

Section: Multi-modal and Cross-modal Retrievalmentioning

confidence: 99%

An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

et al. 2019

View full text Add to dashboard Cite

show abstract

“…In this section, we describe several works related to our work. Some works on the detection of content groups from different social media platforms [2,9,39,40,44,45] are strongly related to the strategy for solving problem 1 (see Section 1). The content groups detected in those works can provide complementary information delivered by multiple platforms, which is much richer information than the information from a single platform.…”

Section: Related Workmentioning

confidence: 99%

“…For example, mining of semantic relationships among multi-modal contents obtained from different social media platforms by fusing two uni-modal graphs, i.e. text and visual graphs, with upload-time similarities [9,45] and user behavior information [39] was performed. In another work, information on hot search queries was used as guidance to calculate similarities between contents of different platforms [40].…”

Section: Related Workmentioning

confidence: 99%

Extracting hierarchical structure of content groups from different social media platforms using multiple social metadata

Takehara

Harakawa

Ogawa

et al. 2017

Multimed Tools Appl

View full text Add to dashboard Cite

A novel scheme for retrieving users' desired contents, i.e., contents with topics in which users are interested, from multiple social media platforms is presented in this paper. In existing retrieval schemes, users first select a particular platform and then input a query into the search engine. If users do not specify suitable platforms for their information needs and do not input suitable queries corresponding to the desired contents, it becomes difficult for users to retrieve the desired contents. The proposed scheme extracts the hierarchical structure of content groups (sets of contents with similar topics) from different social media platforms, and it thus becomes feasible to retrieve desired contents even if users do not specify suitable platforms and do not input suitable queries. This paper has two contributions: (1) A new feature extraction method, Locality Preserving Canonical Correlation Analysis with multiple social metadata (LPCCA-MSM) that can detect content groups without the boundaries of different social media platforms is presented in this paper. LPCCA-MSM uses multiple social metadata as auxiliary information unlike conventional methods that only use content-based information such as textual or visual features. (2) The proposed novel retrieval scheme can realize hierarchical content structuralization from different social media platforms. The extracted hierarchical structure shows various abstraction levels of content groups and their hierarchical relationships, which can help users select topics related to the input query. To the best of our knowledge, an intensive study on such an application has not been conducted; therefore, this paper has strong novelty. To verify the effectiveness of the above contributions, extensive experiments for real-world datasets containing YouTube videos and Wikipedia articles were conducted.

show abstract

“…Document [4] proposed an unsupervised method called convolutional cross Autoencoder for cross-modality element-level feature learning, which can capture the cross-modality correlations in element samples of social media datasets. The reference [5] proposed a multi-modality fusion framework and a topic recovery approach to effectively detect topics from cross-media data. Reference [6] proposed a modality-dependent cross-media retrieval model, where two couples of projections are learned for different CMR tasks instead of one a couple of projections.…”

Section: Introductionmentioning

confidence: 99%

Csrncva: A Model of Cross-Media Semantic Retrieval Based on Neural Computing of Visual and Auditory Sensations

Liu¹,

Cai²,

Li³

et al. 2018

NNW

View full text Add to dashboard Cite

Cross-media semantic retrieval (CSR) and cross-modal semantic mapping are key problems of the multimedia search engine. The cognitive function and neural structure for visual and auditory information process are an important reference for the study of brain-inspired CSR. In this paper, we analyze the hierarchy, the functionality and the structure of visual and auditory in the brain. Considering an idea from deep belief network and hierarchical temporal memory, we presented a brain-inspired intelligent model, called cross-media semantic retrieval based on neural computing of visual and auditory sensation (CSRNCVA). Algorithms based on CSRNCVA were developed. It employs belief propagation algorithms of probabilistic graphical model and hierarchical learning. The experiments show that our model and algorithms can be effectively applied to the CSR. This work provides an important significance for brain-inspired cross-media intelligence framework.

show abstract

Effective Multimodality Fusion Framework for Cross-Media Topic Detection

Cited by 35 publications

References 42 publications

An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

Extracting hierarchical structure of content groups from different social media platforms using multiple social metadata

Csrncva: A Model of Cross-Media Semantic Retrieval Based on Neural Computing of Visual and Auditory Sensations

Contact Info

Product

Resources

About