Spatial Weighting for Bag-of-Visual-Words and Its Application in Content-Based Image Retrieval

Abstract

Abstract. It is a challenging and important task to retrieve images from a large and highly varied image data set based on their visual contents. Problems like how to fill the semantic gap between image features and the user have attracted a lot of attention from the research community. Recently, the 'bag of visual words' approach exhibits very good performance in content-based image retrieval (CBIR). However, since the 'bag of visual words' approach represents an image as an unordered collection of local descriptors which only use the intensity information, the resulting model provides little insight about the spatial constitution and color information of the image. In this paper, we develop a novel image representation method which uses Gaussian mixture model (GMM) to provide spatial weighting for visual words and apply this method to facilitate content based image retrieval. Our approach is a simple and more efficient compared with the order-less 'bag of visual words' approach. In our method, firstly, we extract visual tokens from the image data set and cluster them into a lexicon of visual words. Then, we represent the spatial constitution of an image as a mixture of n Gaussians in the feature space and decompose the image into n regions. The spatial weighting scheme is achieved by weighting visual words according to the probability of each visual word belonging to each of the n regions in the image. The cosine similarity between spatial weighted visual word vectors is used as distance measurement between regions, while the image-level distance is obtained by averaging the pair-wise distances between regions. We compare the performance of our method with the traditional 'bag of visual words' and 'blobworld' approaches under the same image retrieval scenario. Experimental results demonstrate that the our method is able to tell images apart in the semantic level and improve the performance of CBIR.

Discriminative feature fusion for image classification

“…For LRFF, we use the marginalized kernel of Eq. (7). Note that we use the mean average precision (M. AP.…”

“…They characterize each image with histograms of shape visual words (one histogram per concept) in which the frequency of each visual word is weighted by its (color) probability to belong to the considered concept. Likewise, Elsayad et al [10] and Chen et al [7] propose to weight the contribution of each shape visual words in the histogram by using a probability derived from color information. The drawback of these approaches is that the resulting representation is a shape-based histogram, i.e.…”

From Text to Images: Weighting Schemes for Image Retrieval

“…[8] proposed a work similar to the one of Jiang et al, also based on the properties of the clustering rather than statistical properties. More recently, [46] proposed a weighting scheme based on a previous segmentation of the image based on color, to integrate geometrical information to their system. Here again, the statistical properties of visual words in the image and in the collection is not used.…”

Intensity and affine invariant statistic-based image matching

“…Features such as color and texture were discarded for the reasons discussed in Section 1. In this context we have tried to apply the state of the art in the extraction of local information from the images using the Bag of Visual Words (BOVW) (Yang et al 2007;Chen et al 2009;Csurka et al 2004). We used the SURF algorithm as a feature descriptor and we have performed a trial and error test to select 150 as the best number of words for the BOVW dictionary.…”