Content-Based Multimedia Retrieval Using Feature Correlation Clustering and Fusion

Abstract

Nowadays, only processing visual features is not enough for multimedia semantic retrieval due to the complexity of multimedia data, which usually involve a variety of modalities, e.g. graphics, text, speech, video, etc. It becomes crucial to fully utilize the correlation between each feature and the target concept, the feature correlation within modalities, and the feature correlation across modalities. In this paper, the authors propose a Feature Correlation Clustering-based Multi-Modality Fusion Framework (FCC-MMF) for multimedia semantic retrieval. Features from different modalities are combined into one feature set with the same representation via a normalization and discretization process. Within and across modalities, multiple correspondence analysis is utilized to obtain the correlation between feature-value pairs, which are then projected onto the two principal components. K-medoids algorithm, which is a widely used partitioned clustering algorithm, is selected to minimize the Euclidean distance within the resulted clusters and produce high intra-correlated feature-value pair clusters. Majority vote is applied to subsequently decide which cluster each feature belongs to. Once the feature clusters are formed, one classifier is built and trained for each cluster. The correlation and confidence of each classifier are considered while fusing the classification scores, and mean average precision is used to evaluate the final ranked classification scores. Finally, the proposed framework is applied on NUS-wide Lite data set to demonstrate the effectiveness in multimedia semantic retrieval.

Building Multi-model Collaboration in Detecting Multimedia Semantic Concepts

“…In the training section, the first three steps, e.g., preprocessing, feature-value pair projection, and feature-value pair clustering, are the same processes as proposed in our previous work FCC-MMF [6]. In the next highlighted green square, a new factor γ is introduced to refine ARC fusion method proposed in [7].…”

“…In the second set of experiments, the proposed MMC framework is also compared against our previous works, e.g. (CFA-MMF) [26] and (FCC-MMF) [6], and the original flat concatenation of multi-modality features, which is exact the same feature set but it is only trained as one classifier and there is no fusion process involved, on NUS-WIDE-LITE dataset.…”

“…In this paper, built on our previous works [6][7], we proposed a Multi-Model Collaboration (MMC) framework for multimedia semantic concept detection by introducing a novel cluster-based ARC fusion method, where ARC stands for adjustment, reliability and correlation of the intervals to the target semantic concept. In our previous works, features extracted from multiple modalities are transformed into feature clusters with high intra-correlation and low inter-correlation.…”

Gaussian Mixture Model-Based Subspace Modeling for Semantic Concept Retrieval

“…Nowadays, with the ubiquitous Internet and the prosperity of social media, people upload numerous images and videos to their personal online repositories to share with their families and friends frequently. The contentbased retrieval methods [1] [2][3] [4][5] [6][7] [8] have achieved great success for various applications in the past decades though they still suffer from the so-called semantic gap issue [8] [9]. Researchers have proposed a lot of multimedia contentbased retrieval approaches to bridge the semantic gaps and to enhance the retrieval performance in multimedia research [10] [11] [12][13] [14][15] [16] [17] [18] [19] [20].…”

Integrating Deep Learning with Correlation-based Multimedia Semantic Concept Detection

“…Originally, MCA was extended from the standard correspondence analysis to analyze the correlation among variables. Later, it has demonstrated its competence in enhancing multimedia retrieval research topics through capturing the correlations among high-level semantic concepts and low-level features [74,111,116], and modeling posterior probability [73,110,134].…”