Towards optimal bag-of-features for object categorization and semantic video retrieval

Jiang, Yu‐Gang; Ngo, Chong-Wah; Yang, Jun

doi:10.1145/1282280.1282352

Cited by 538 publications

(383 citation statements)

References 15 publications

Supporting

Mentioning

369

Contrasting

Unclassified

Order By: Relevance

“…In recent study, there has been an intense focus on applying term weighting schemes (like tf, idf) to the 'bag-of-visual-words' feature vectors [4,6] . Extensive study in [15] suggests that when the vocabulary size of visual words is around 1000, the tf-idf weighting performs best. Recall that in our approach, the vocabulary size of visual words for the data collection is D=1000, to make the experimental results comparable, we chose to compare our spatial weighting approach with the tf-idf weighted 'bag-of-visual-words' approach [4] .…”

Section: Resultsmentioning

confidence: 99%

“…Many effective text mining and information retrieval algorithms like tf-idf weighting, stop word removal and feature selection have been applied to the vector-space model of visual-words. Problems such as how vocabulary size and term weighting schemes affect the performance of 'bag-of-visualwords' representation are also studied in recent research works [4,15] .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Spatial Weighting for Bag-of-Visual-Words and Its Application in Content-Based Image Retrieval

Chen

Shen

2009

Advances in Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Abstract. It is a challenging and important task to retrieve images from a large and highly varied image data set based on their visual contents. Problems like how to fill the semantic gap between image features and the user have attracted a lot of attention from the research community. Recently, the 'bag of visual words' approach exhibits very good performance in content-based image retrieval (CBIR). However, since the 'bag of visual words' approach represents an image as an unordered collection of local descriptors which only use the intensity information, the resulting model provides little insight about the spatial constitution and color information of the image. In this paper, we develop a novel image representation method which uses Gaussian mixture model (GMM) to provide spatial weighting for visual words and apply this method to facilitate content based image retrieval. Our approach is a simple and more efficient compared with the order-less 'bag of visual words' approach. In our method, firstly, we extract visual tokens from the image data set and cluster them into a lexicon of visual words. Then, we represent the spatial constitution of an image as a mixture of n Gaussians in the feature space and decompose the image into n regions. The spatial weighting scheme is achieved by weighting visual words according to the probability of each visual word belonging to each of the n regions in the image. The cosine similarity between spatial weighted visual word vectors is used as distance measurement between regions, while the image-level distance is obtained by averaging the pair-wise distances between regions. We compare the performance of our method with the traditional 'bag of visual words' and 'blobworld' approaches under the same image retrieval scenario. Experimental results demonstrate that the our method is able to tell images apart in the semantic level and improve the performance of CBIR.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Spatial Weighting for Bag-of-Visual-Words and Its Application in Content-Based Image Retrieval

Chen

Shen

2009

Advances in Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

show abstract

“…In particular, so-called "bag-of-features" models have become very popular, where the computation of local image descriptors is initiated by either scale invariant or affine invariant interest points (Sivic et al [145], Nowak et al [33], Jiang et al [59]); see also Mikolajczyk et al [126] for an experimental evaluation of image descriptors at interest points, Moreels and Perona [127] and Aanaes et al [1] for evaluations of feature detectors and image descriptors on three-dimensional datasets and Kaneva et al [65] for evaluations using photorealistic virtual worlds. For all of these recognition approaches, the invariance properties of the recognition system rely heavily on the invariance properties of the interest points at which local image descriptors are computed.…”

Section: Related Workmentioning

confidence: 99%

Image Matching Using Generalized Scale-Space Interest Points

Lindeberg

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The performance of matching and object recognition methods based on interest points depends on both the properties of the underlying interest points and the choice of associated image descriptors. This paper demonstrates advantages of using generalized scale-space interest point detectors in this context for selecting a sparse set of points for computing image descriptors for image-based matching. For detecting interest points at any given scale, we make use of the Laplacian ∇ 2 norm L, the determinant of the Hessian det H norm L and four new unsigned or signed Hessian feature strength measures D 1,norm L,D 1,norm L, D 2,norm L and D 2,norm L, which are defined by generalizing the definitions of the Harris and Shi-and-Tomasi operators from the second moment matrix to the Hessian matrix. Then, feature selection over different scales is performed either by scale selection from local extrema over scale of scale-normalized derivates or by linking features over scale into feature trajectories and computing a significance measure from an integrated measure of normalized feature strength over scale. A theoretical analysis is presented of the robustness of the differential entities underlying these interest points under image deformations, in terms of invariance properties under affine image deformations or approximations thereof. Disregarding the effect of the rotationally symmetric scale-space smoothing operation, the determinant of the Hessian det H norm L is a truly affine covariant differential entity and the Hessian feature strength measures D 1,norm L andD 1,norm L have a major contribution from the affine covariant determinant of the Hessian, implying that local extrema of these differen- It is shown how these generalized scale-space interest points allow for a higher ratio of correct matches and a lower ratio of false matches compared to previously known interest point detectors within the same class. The best results are obtained using interest points computed with scale linking and with the new Hessian feature strength measures D 1,norm L,D 1,norm L and the determinant of the Hessian det H norm L being the differential entities that lead to the best matching performance under perspective image transformations with significant foreshortening, and better than the more commonly used Laplacian operator, its difference-of-Gaussians approximation or the Harris-Laplace operator. We propose that these generalized scale-space interest points, when accompanied by associated local scale-invariant image descriptors, should allow for better performance of interest point based methods for image-based matching, object recognition and related visual tasks.

show abstract

“…The image signature then can be seen as a histogram of occurred visual words. The works ( [5], [6]) reflected the fact that the quantization effect provides a very coarse approximation to the actual distance between two features -zero if assigned to the same visual word and infinite otherwise. Such approach is called hard assignment.…”

Section: Visual Words Assignmentmentioning

confidence: 99%

“…The work [5] compares several detectors of local image features and evaluate their performance on visual codebooks with different sizes. The explored detectors are time expensive and the codebook sizes are only up to 10k.…”

Section: Introductionmentioning

confidence: 99%

Visual Codebooks Survey for Video On-Line Processing

Beran

Zemčík

2010

Computer Vision and Graphics

View full text Add to dashboard Cite

Abstract. This paper explores techniques in the pipeline of image description based on visual codebooks suitable for video on-line processing. The pipeline components are (i) extraction and description of local image features, (ii) translation of each high-dimensional feature descriptor to several most appropriate visual words selected from the discrete codebook and (iii) combination of visual words into bag-of-words using hard or soft assignment weighting scheme. For each component, several state-of-the-art techniques are analyzed and discussed and their usability for video on-line processing is addressed. The experiments are evaluated on the standard Kentucky and Oxford building datasets using image retrieval framework. The results show the impact loosing the pipeline precision in the price of improving the time cost which is crucial for real-time video processing.

show abstract

Towards optimal bag-of-features for object categorization and semantic video retrieval

Cited by 538 publications

References 15 publications

Spatial Weighting for Bag-of-Visual-Words and Its Application in Content-Based Image Retrieval

Spatial Weighting for Bag-of-Visual-Words and Its Application in Content-Based Image Retrieval

Image Matching Using Generalized Scale-Space Interest Points

Visual Codebooks Survey for Video On-Line Processing

Contact Info

Product

Resources

About