NLP for indexing and retrieval of captioned photographs

Common image-text joint understanding techniques presume that images and the associated text can universally be characterized by a single implicit model. However, co-occurring images and text can be related in qualitatively different ways, and explicitly modeling it could improve the performance of current joint understanding models. In this paper, we train a Cross-Modal Coherence Model for text-to-image retrieval task. Our analysis shows that models trained with image–text coherence relations can retrieve images originally paired with target text more often than coherence-agnostic models. We also show via human evaluation that images retrieved by the proposed coherence-aware model are preferred over a coherence-agnostic baseline by a huge margin. Our findings provide insights into the ways that different modalities communicate and the role of coherence relations in capturing commonsense inferences in text and imagery.

show abstract

Image Retrieval Through Free-Form Query using Intelligent Text Processing

Angadi¹,

Purad²

2023

IJITEE

View full text Add to dashboard Cite

Image Retrieval is the process of retrieving images from the image/multimedia databases. Retrieval of images are carried out with various types of queries, free-form query is a text-query that consists of single or multiple keywords and/or concepts or descriptions of images with or without the inclusion of wild-card characters and/or punctuations. This work aims to handle image retrieval based on free-form text queries. Simple & complex queries of conceptual descriptions of images are explored and an intelligent processing system with free-form queries based on the Bag-of-Words model is modified and built for natural scene images and on Diverse Social Images using the Damerau-Levenshtein edit distance measure. The efficacy of the proposed system is evaluated by testing 1500 free-form text queries and has resulted in a recall accuracy of 91.3% on natural scene images (of Wang/Corel database) and 100% on Diverse Social Images (of DIV400 dataset). These results show that the system proposed has produced satisfactory performance compared to published results such as the harmonic mean of precision and recall (i.e. F1-Score) of 76.70% & 63.32% at retrieval of 20 images etc in reported works.

show abstract

NLP for indexing and retrieval of captioned photographs

Cited by 3 publications

References 7 publications

Automatic video description generation via LSTM with joint two-stream encoding

Automatic video description generation via LSTM with joint two-stream encoding

Cross-Modal Coherence for Text-to-Image Retrieval

Image Retrieval Through Free-Form Query using Intelligent Text Processing

Contact Info

Product

Resources

About