A Cross-media Model for Automatic Image Annotation

Abstract

Automatic image annotation is still an important open problem in multimedia and computer vision. The success of media sharing websites has led to the availability of large collections of images tagged with human-provided labels. Many approaches previously proposed in the literature do not accurately capture the intricate dependencies between image content and annotations. We propose a learning procedure based on Kernel Canonical Correlation Analysis which finds a mapping between visual and textual words by projecting them into a latent meaning space. The learned mapping is then used to annotate new images using advanced nearest-neighbor voting methods. We evaluate our approach on three popular datasets, and show clear improvements over several approaches relying on more standard representations.

Fisher Encoded Convolutional Bag-of-Windows for Efficient Image Retrieval and Social Image Tagging

“…The MIRFLICKR-25K dataset [15] is composed of 25,000 images from Flickr with 1,386 user tags that occur in at least 20 images, and is split in 12,500 for training and 12,500 for testing, with exactly the same partition as [4,12]. In addition ground truth annotations for 18 tags are provided on the whole set.…”

“…As low-level features are hardly semantically related, Guillaumin et al [12] and Verma et al [30] proposed to learn a weighted metric to improve on precision. Ballan et al [4] proposed using KCCA to learn mid-level features to be used with previous nearest neighbors approaches.…”

“…The goal of the second task is to predict, for a given image, a finite set of tags from a given vocabulary, serving as a compact description of the image. A popular group of recent image annotation methods apply tag propagation using diversely defined image neighborhoods [4,12,[19][20][21]30]. These approaches have been successfully applied to the context of social and user generated media, that are typically annotated with tags that are likely to correlate with image content.…”

Adaptive Tag Selection for Image Annotation

“…While the number of tags that can be modeled is increasing, it remains unclear how to select a proper number of tags to annotate an unlabeled image. This problem is overlooked, because most of the existing works either assess the top-k ranked tags [3,4] or the entire tag ranking list [6].…”

“…Quite a few methods have been proposed for image annotation, either by building visual classifiers per tag [1,2] or by propagating tags from visually similar images [3,4]. Given a novel image and a tag vocabulary, these methods first compute each tag's relevance score with respect to the given image, and sort the tags in descending order by their scores.…”