Large Scale Online Learning of Image Similarity through Ranking

Abstract

Learning a measure of similarity between pairs of objects is an important generic problem in machine learning. It is particularly useful in large scale applications like searching for an image that is similar to a given image or finding videos that are relevant to a given video. In these tasks, users look for objects that are not only visually similar but also semantically related to a given object. Unfortunately, the approaches that exist today for learning such semantic similarity do not scale to large datasets. This is both because typically their CPU and storage requirements grow quadratically with the sample size, and because many methods impose complex positivity constraints on the space of learned similarity functions.The current paper presents OASIS, an Online Algorithm for Scalable Image Similarity learning that learns a bilinear similarity measure over sparse representations. OASIS is an online dual approach using the passive-aggressive family of learning algorithms with a large margin criterion and an efficient hinge loss cost. Our experiments show that OASIS is both fast and accurate at a wide range of scales: for a dataset with thousands of images, it achieves better results than existing state-of-the-art methods, while being an order of magnitude faster. For large, web scale, datasets, OASIS can be trained on more than two million images from 150K text queries within 3 days on a single CPU. On this large scale dataset, human evaluations showed that 35% of the ten nearest neighbors of a given test image, as found by OASIS, were semantically relevant to that image. This suggests that query independent similarity could be accurately learned even for large scale datasets that could not be handled before.

A Latent Variable Ranking Model for Content-Based Retrieval

“…Obtaining ground truth for training and testing retrieval algorithms is a challenging task. In the absence of feedback from real users of a retrieval system, alternative approaches have been proposed to obtain ground truth similarity data (e.g., [5]). In this paper we use semantic annotations available in the datasets to generate ground truth similarities, as will be described next.…”

“…4 The mixture model outperforms the global and transductive models in all cases. 5 To give a concrete example, in the known-database setting for the SUN data, for a recall of 20% the global and transductive models obtain around 7% precision versus 9% precision of the mixture model. This means that to obtain 20 of the top 100 neighbors, on average 286 images of the 6,000 database images need to be browsed with the global and transductive models.…”

“…These types of constraints could be obtained from feedback of users of the retrieval system. In the standard ranking SVMs [18,1,9,8,5], one assumes that a set of elementary pair-wise similarity functions is given and uses the triplets to learn an optimal weigthed combination of these functions. In this global ranking model the same weighted combination is used for all queries independently of where they lie in the query space.…”

“…Chechik et al [5] presented an online dual algorithm to find an optimal Z using ranking constraints. The loss function used in [5] is a generalization of the hinge loss for the ranking setting, which is the same loss function used in this paper. Finally, Lanckriet et al [13] proposed a method to learn a global linear combination of predefined kernel functions.…”

Latent topics-based relevance feedback for video retrieval

“…The standard CBVR procedure involves three main components: (i) a query, containing a few video examples of the semantic concept that the user is looking for; (ii) a database, which is used to retrieve videos related to the query concept; and (iii) a ranking function, which sorts the database according to the relevance with respect to the user's query. These three components are typically integrated with the user in a Relevance Feedback (RF) scheme [5] to provide the most relevant videos through several feedback iterations. Figure 1 shows the general RF scheme for retrieval.…”