Multi-Modal Image Retrieval for Complex Queries using Small Codes

Abstract

We propose a unified framework for image retrieval capable of handling complex and descriptive queries of multiple modalities in a scalable manner. A novel aspect of our approach is that it supports query specification in terms of objects, attributes and spatial relationships, thereby allowing for substantially more complex and descriptive queries. We allow these complex queries to be specified in three di↵erent modalities -images, sketches and structured textual descriptions. Furthermore, we propose a unique multi-modal hashing algorithm capable of mapping queries of di↵erent modalities to the same binary representation, enabling e cient and scalable image retrieval based on multi-modal queries. Extensive experimental evaluation shows that our approach outperforms the state-of-the-art image retrieval and hashing techniques on the MSRC and SUN09 datasets by about 100%, while the performance on a dataset of 1M images, from Flickr, demonstrates its scalability.

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

“…Some other methodologies are Partial Least Squares (PLS) and correlation matching. With respect to the former, a PLS-based framework, which maps queries from multiple modalities to points in a common linear subspace, is introduced in [24]. Regarding the latter, [21] utilizes correlation matching between the textual and visual modalities of multimedia documents in the task of cross-modal document retrieval.…”

“…The Partial Least Squares (PLS) model has been used in multimodal fusion [24] to efficiently combine two modalities. Given two matrices X and Y , PLS decomposes X and Y as follows:…”

“…Our method extends our previous work [8] using Partial Least Squares (PLS) Regression to combine multiple views of the same modality, such as SIFT descriptors and visual features based on Deep Convolution Neural Networks. The proposed method is motivated by the PLS approach [24], due to its effectiveness in multimodal hashing, and is compared to several baseline methods in unsupervised multimedia retrieval, such as weighted linear, non-linear, diffusion-based and advanced graph-based models. We also take into account complexity issues when handling several forms of data and metadata and our framework's memory complexity is comparable to a bi-modal multimedia retrieval framework.…”

Retrieval of Images with Objects of Specific Size, Location, and Spatial Configuration

“…The authors of [12] and [27] investigate image retrieval with structured object queries by encoding objectnames and certain relations among objects with textual phrases like "car on road". While the query in [12] is restricted to queries with word descriptions (such as Q = {car on road}), our approach can be applied to cases where the queries can not necessarily be represented by textual descriptions.…”

A new indexing method for complex similarity queries in immersive multimedia systems

“…The discriminating power of the CBMR system may increase as more information describing the media content is included in the metadata. Moreover, the multimodal CBMR system can capture users' subjective query concepts thoroughly, and hence achieve high search accuracy [23,29]. In this paper, we assume that complex similarity queries are composed of Boolean combinations of feature descriptions.…”