Generating Image Descriptions Using Semantic Similarities in the Output Space

Abstract

Automatically generating meaningful descriptions for images has recently emerged as an important area of research. In this direction, a nearest-neighbour based generative phrase prediction model (PPM) proposed by (Gupta et al. 2012) was shown to achieve state-of-the-art results on PASCAL sentence dataset, thanks to the simultaneous use of three different sources of information (i.e. visual clues, corpus statistics and available descriptions). However, they do not utilize semantic similarities among the phrases that might be helpful in relating semantically similar phrases during phrase relevance prediction. In this paper, we extend their model by considering inter-phrase semantic similarities. To compute similarity between two phrases, we consider similarities among their constituent words determined using WordNet. We also re-formulate their objective function for parameter learning by penalizing each pair of phrases unevenly, in a manner similar to that in structured predictions. Various automatic and human evaluations are performed to demonstrate the advantage of our "semantic phrase prediction model" (SPPM) over PPM.

A Probabilistic Approach for Image Retrieval Using Descriptive Textual Queries

“…These descriptions are usually in the form of simple captions containing few tens of words. Among these, there are two popular practices: either to generate a description given an image [6,16,17], or to retrieve one from a collection of available descriptions [11,14,18]. In the first setting, a new description is generated by combining visual clues using natural language generation (NLG) techniques.…”

“…This in turn can be helpful in analyzing the interplay among the individual components of the query. Conceptually, our work closely relates with the image description generation methods [6,17], and demonstrates their application to the image retrieval task given descriptive textual queries.…”

“…where Zsim(yi, yj) ∈ [0, 1] denotes similarity between the phrases yi and yj. This is computed using the procedure described in [17], which is based on WordNet based similarity between the individual terms of the two phrases. The above definition denotes that if the phrase yi is present in the ground-truth phrases of J, then δy i ,J = 1.…”

An image caption model based on attention mechanism and deep reinforcement learning

“…The search-based method involves using similarity algorithms to compute the similarity between extracted features and the images stored in a constructed image library, to find out the images in line with the algorithm, and these images have been matched with the corresponding sentence descriptions in advance, which can be fine-tuned for appropriate output. Verma et al (2013) adopt traditional image feature extraction methods to compare the extracted image features with those in the database, so as to determine the maximum joint probability output in the description tuple. Li and Jin (2016) introduce the reordering mechanism which greatly improves the model performance.…”