Web media semantic concept retrieval via tag removal and model fusion

Abstract

Multimedia data on social websites contain rich semantics and are often accompanied with user-defined tags. To enhance Web media semantic concept retrieval, the fusion of tag-based and content-based models can be used, though it is very challenging. In this article, a novel semantic concept retrieval framework that incorporates tag removal and model fusion is proposed to tackle such a challenge. Tags with useful information can facilitate media search, but they are often imprecise, which makes it important to apply noisy tag removal (by deleting uncorrelated tags) to improve the performance of semantic concept retrieval. Therefore, a multiple correspondence analysis (MCA)-based tag removal algorithm is proposed, which utilizes MCA's ability to capture the relationships among nominal features and identify representative and discriminative tags holding strong correlations with the target semantic concepts. To further improve the retrieval performance, a novel model fusion method is also proposed to combine ranking scores from both tag-based and content-based models, where the adjustment of ranking scores, the reliability of models, and the correlations between the intervals divided on the ranking scores and the semantic concepts are all considered. Comparative results with extensive experiments on the NUS-WIDE-LITE as well as the NUS-WIDE-270K benchmark datasets with 81 semantic concepts show that the proposed framework outperforms baseline results and the other comparison methods with each component being evaluated separately.

Correlation-based Feature Analysis and Multi-Modality Fusion framework for multimedia semantic retrieval

“…Liu et al [7] propose a method called Selective Weighted Late Fusion (SWLF) which used the results trained from a binary classifier to weight the corresponding features in testing data set. Chen et al [8] propose a fusion strategy to combine ranking scores from both tag-based and content-based models, where the adjustment, reliability, and correlation of ranking scores from different models are all considered. Hofmann et al [9] propose a fusion method based on probabilistic kernel density estimation to fuse the output of part-based object detectors from multiple camera views in person detection.…”

“…The final fusion of the scores from multiple models are based on the refined fusion scheme ARC [8] expressed as…”

“…Specifically, it is calculated as the average precision of the m th model evaluated on the instances in the training set; θ denotes the relationship between the testing score for the m th model and the target concept, which is measured based on the correlation value between the testing score interval and the related concept [8]; ω m is a scale factor to balance the ranking score for the m th model, which is refined in this paper by using the absolute mean score for all the training instances.…”

“…The dataset provides the ground truth for 79 concepts and several low-level features commonly used for evaluation such as 64-dimensional color histogram and 128-dimensional wavelet texture, which were also used in this paper. In addition, we also utilize the textual features extracted using the method proposed in [8].…”

“…Fig. 2 shows the number of features after applying the proposed framework (denoted as CFA-MMF) and the original feature set including both visual and textual features (with noisy tag removed [8]). With the proposed correlationbased feature analysis method, only features with high correlation toward other features will be used in classification process.…”