Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

Abstract

The ability to associate images with natural language sentences that describe what is depicted in them is a hallmark of image understanding, and a prerequisite for applications such as sentence-based image search. In analogy to image search, we propose to frame sentence-based image annotation as the task of ranking a given pool of captions. We introduce a new benchmark collection for sentence-based image description and search, consisting of 8,000 images that are each paired with five different captions which provide clear descriptions of the salient entities and events. We introduce a number of systems that perform quite well on this task, even though they are only based on features that can be obtained with minimal supervision. Our results clearly indicate the importance of training on multiple captions per image, and of capturing syntactic (word order-based) and semantic features of these captions. We also perform an in-depth comparison of human and automatic evaluation metrics for this task, and propose strategies for collecting human judgments cheaply and on a very large scale, allowing us to augment our collection with additional relevance judgments of which captions describe which image. Our analysis shows that metrics that consider the ranked list of results for each query image or sentence are significantly more robust than metrics that are based on a single response per query. Moreover, our study suggests that the evaluation of ranking-based image description systems may be fully automated.

Leveraging High Level Visual Information for Matching Images and Captions

“…We show that the combination of the basic BoW text kernel and a high level image kernel based on the probabilities given by visual classifiers outperforms the best combination in [17] in most evaluation metrics (Section 5.1); -We demonstrate that visual classifiers trained for synsets included in the evaluation dataset improve the retrieval scores by a factor of two compared to the best method in [17] (Section 5.2); -Finally, in contrast to lexical similarities computed using text corpora, we propose to use the high level visual information to learn a lexical similarity, and show that the BoW text kernel enriched with such lexical similarity further boosts the performance (Section 6).…”

“…In [17] the problem of generating natural language description for a given image is relaxed to one of ranking a set of humanwritten captions, by assuming the set contains the original (human-written) caption of the image. [17] builds a dataset (dubbed Flickr8K) of image and caption pairs, and employs the kernel canonical correlation analysis (KCCA) [18,19] to learn a latent space in which a similarity measure between an image and a caption is defined. KCCA requires two kernels to be built, one for the images and the other for the captions.…”

“…KCCA requires two kernels to be built, one for the images and the other for the captions. [17] fixes the image kernel to a relatively simple one that uses only low level and mid-level visual information such as colour, texture and SIFT descriptors, and demonstrates that text kernels that exploit lexical similarities and high-order co-occurrence information outperform the basic Bagof-Word (BoW) text kernel.…”

“…In this paper, we build on the results from [17], and propose an approach that significantly improves the performance of image-to-text annotation and text-toimage retrieval. In particular we consider the scenario where there is no image data for training visual classifiers for synsets in the application domain at hand.…”

“…In particular, we present the dataset and experimental setup introduced in [17], which we compare our approaches to. The BoW text kernel used in our studies is presented in Section 3, followed by an introduction in Section 4 to our visual recognition pipeline that is based on a deep convolutional neural network (CNN).…”