Improving image annotation via ranking‐oriented neighbor search and learning‐based keyword propagation

Abstract

Automatic image annotation plays a critical role in modern keyword-based image retrieval systems. For this task, the nearest-neighbor-based scheme works in two phases: first, it finds the most similar neighbors of a new image from the set of labeled images; then, it propagates the keywords associated with the neighbors to the new image. In this article, we propose a novel approach for image annotation, which simultaneously improves both phases of the nearest-neighbor-based scheme. In the phase of neighbor search, different from existing work discovering the nearest neighbors with the predicted distance, we introduce a ranking-oriented neighbor search mechanism (RNSM), where the ordering of labeled images is optimized directly without going through the intermediate step of distance prediction. In the phase of keyword propagation, different from existing work using simple heuristic rules to select the propagated keywords, we present a learning-based keyword propagation strategy (LKPS), where a scoring function is learned to evaluate the relevance of keywords based on their multiple relations with the nearest neighbors. Extensive experiments on the Corel 5K data set and the MIR Flickr data set demonstrate the effectiveness of our approach.

Cross-Media Similarity Evaluation for Web Image Retrieval in the Wild

“…The study by Cheng et al [42] empirically investigates the effects of multiple information evidence on social image retrieval, where a query consists of a query tag and an example image to facilitate different retrieval strategies. To attack the unreliability of social image tagging, Cui et al [43] introduce a supervision step into the neighbor voting scheme [44] to make the neighbors reweighted towards optimizing the ranking performance of tag-based image retrieval, while Cui et al [45] improve neighbor voting by fusing multiple visual features. Besides tag-based image retrieval, we go a step further by considering real-user queries from a commercial web image search engine.…”

Learning to rank images for complex queries in concept-based search

“…Since the indicator function 1 (·) is nonsmooth, directly optimizing the empirical risk in Eq. (8) is computationally infeasible [14]. To address the problem, we adopt the Ranking SVM framework [17] as the backbone of our learning method.…”

“…Intuitively, through the supervision step, the possibility is offered that utilizing the information from the data collection to steer the search process and reduce the need for making heuristic assumptions [13]. Although great success has been achieved [14,15], few research efforts have been devoted to exploring the potential of learning to rank in concept-based image search.…”

PSI: A probabilistic semantic interpretable framework for fine‐grained image ranking

“…Image ranking is getting more and more important and sometimes indispensable in current information applications. For instance, a web search engine can apply ranking techniques to optimize the order of retrieved images candidates based on certain user-specified criteria, thereby improving the user experience (Cui, Ma, Lian, Chen, & Wang, 2015;Deng, Ji, Tao, Gao, & Li, 2014;Morioka & Wang, 2011;Yang & Hanjalic, 2010). Besides, image ranking also has attracted the attention of various research communities, such as computer vision, pattern recognition and machine learning, extensive research has been conducted on this topic (Grangier & Bengio, 2008;Chechik, Sharma, Shalit, & Bengio, 2010;Huang, Feris, Chen, & Yan, 2015;Escorcia, Niebles, & Ghanem, 2015) in the past decades.…”