Late fusion of heterogeneous methods for multimedia image retrieval

Hérnadez, Carlos A.; Sucar, L. Enrique; Montes, Manuel

doi:10.1145/1460096.1460125

Cited by 63 publications

(50 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These are extensions of the classic unimodal systems, where a common retrieval system integrates information from various modalities. This can be done by fusing features from different modalities into a single vector [37], [38], [39], or by learning different models for different modalities and fusing their predictions [40], [41]. One popular approach is to concatenate features from different modalities and rely on unsupervised structure discovery algorithms, such as latent semantic analysis, to find multimodal statistical regularities.…”

Section: Previous Workmentioning

confidence: 99%

“…One popular approach is to concatenate features from different modalities and rely on unsupervised structure discovery algorithms, such as latent semantic analysis, to find multimodal statistical regularities. A good overview of these methods is given in [39], which also discusses the combination of unimodal and multimodal retrieval systems. Multimodal integration has also been applied to retrieval tasks including audiovisual content [42], [43].…”

Section: Previous Workmentioning

confidence: 99%

See 1 more Smart Citation

On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval

Pereira

Coviello

Doyle

et al. 2014

IEEE Trans. Pattern Anal. Mach. Intell.

408

213

View full text Add to dashboard Cite

Abstract-The problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts. A mathematical formulation is proposed, equating the design of cross-modal retrieval systems to that of isomorphic feature spaces for different content modalities. Two hypotheses are then investigated regarding the fundamental attributes of these spaces. The first is that low-level cross-modal correlations should be accounted for. The second is that the space should enable semantic abstraction. Three new solutions to the cross-modal retrieval problem are then derived from these hypotheses: correlation matching (CM), an unsupervised method which models cross-modal correlations, semantic matching (SM), a supervised technique that relies on semantic representation, and semantic correlation matching (SCM), which combines both. An extensive evaluation of retrieval performance is conducted to test the validity of the hypotheses. All approaches are shown successful for text retrieval in response to image queries and vice versa. It is concluded that both hypotheses hold, in a complementary form, although evidence in favor of the abstraction hypothesis is stronger than that for correlation.

show abstract

Section: Previous Workmentioning

confidence: 99%

Section: Previous Workmentioning

confidence: 99%

On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval

Pereira

Coviello

Doyle

et al. 2014

IEEE Trans. Pattern Anal. Mach. Intell.

408

213

View full text Add to dashboard Cite

show abstract

“…), more and more multimedia contents are published on the web by people. However, the explosion of multimedia contents has not been matched by an equivalent increase in the sophistication of multimedia content retrieval technology [7,24,25]. Nowadays, the dominate search engines for multimedia retrieval, such as Google and Bing, are still text-based.…”

Section: Introductionmentioning

confidence: 99%

“…However, these works do not consider the semantic correlation among different media types. Generally, they can be viewed as uni-media retrieval, in which the query example and the retrieved results are of the same media type [7].…”

Section: Introductionmentioning

confidence: 99%

“…Figure 1 illustrates the following example, given an image of a horse, the cross media retrieval model should return all the contents related to the horse concept, e.g., the sound of horse, the introduction text. To address the cross media retrieval problem, advances have been reported over the last decades [7,26,28]. These methods focus on two traditional ways to design cross media retrieval systems: (a) fusing features from different media data into a single vector [23,33]; (b) learning different models for different media data and fusing their outputs [14,32].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Parallel field alignment for cross media retrieval

Mao

Lin

Cai

et al. 2013

Proceedings of the 21st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Cross media retrieval systems have received increasing interest in recent years. Due to the semantic gap between lowlevel features and high-level semantic concepts of multimedia data, many researchers have explored joint-model techniques in cross media retrieval systems. Previous joint-model approaches usually focus on two traditional ways to design cross media retrieval systems: (a) fusing features from different media data; (b) learning different models for different media data and fusing their outputs. However, the process of fusing features or outputs will lose both low-and highlevel abstraction information of media data. Hence, both ways do not really reveal the semantic correlations among the heterogeneous multimedia data. In this paper, we introduce a novel method for the cross media retrieval task, named Parallel Field Alignment Retrieval (PFAR), which integrates a manifold alignment framework from the perspective of vector fields. Instead of fusing original features or outputs, we consider the cross media retrieval as a manifold alignment problem using parallel fields. The proposed manifold alignment algorithm can effectively preserve the metric of data manifolds, model heterogeneous media data and project their relationship into intermediate latent semantic spaces during the process of manifold alignment. After the alignment, the semantic correlations are also determined. In this way, the cross media retrieval task can be resolved by the determined semantic correlations. Comprehensive experimental results have demonstrated the effectiveness of our approach.

show abstract