Improving web image search results using query-relative classifiers

Abstract

Figure 1. Top-ranked images for the query 'eiffel tower', using (a) a text-based image search engine and (b) our model, ordered left-to-right. AbstractWeb image search using text queries has received considerable attention. However, current state-of-the-art approaches require training models for every new query, and are therefore unsuitable for real-world web search applications. The key contribution of this paper is to introduce generic classifiers that are based on query-relative features which can be used for new queries without additional training. They combine textual features, based on the occurence of query terms in web pages and image meta-data, and visual histogram representations of images. The second contribution of the paper is a new database for the evaluation of web image search algorithms. It includes 71478 images returned by a web search engine for 353 different search queries, along with their meta-data and ground-truth annotations. Using this data set, we compared the image ranking performance of our model with that of the search engine, and with an approach that learns a separate classifier for each query. Our generic models that use query-relative features improve significantly over the raw search engine ranking, and also outperform the query-specific models.

Harvesting Training Images for Fine-Grained Object Categories Using Visual Descriptions

“…Like [2,7,8,12], our work ranks images by their surrounding text. However, we tackle fine-grained object categories which will allow the harvesting of training images to scale to a large number of categories.…”

“…They demonstrated superior results to [2] on the same dataset without requiring any human supervision. George et al [7] build on [12] by retrieving images iteratively, while Krapac et al [8] add contextual features (words surrounding the image etc.) on top of the binary features of [12].…”

“…Previous work [2,7,8,12] performs image-text correspondence by parsing the HTML source code of a web page, and extracting any non-HTML text surrounding an image link, assuming that such text is positioned close to the image. However, this assumption is not always correct as the HTML source does not always dictate how a web page is displayed.…”

“…neighbouring text and metadata information, are useful as well [2,7,8,12] (see Sect. 1.1 for an in-depth discussion).…”

“…Thus, previous work [2,3,6,7,8,9,12,14] has automatically harvested image datasets by retrieving images from online search engines. These images can then be used as training examples for a visual classifier.…”