A system for large-scale image and video retrieval on everyday scenes

Abstract

There has been a growing amount of multimedia data generated on the web todayin terms of size and diversity. This has made accurate content retrieval with these large and complex collections of data a challenging problem. Motivated by the need for systems that can enable scalable and efficient search, we propose QIK (Querying Images Using Contextual Knowledge). QIK leverages advances in deep learning (DL) and natural language processing (NLP) for scene understanding to enable large-scale multimedia retrieval on everyday scenes with common objects. The system consists of three major components: Indexer, Query Processor, and Video Processor. Given an image, the Indexer performs probabilistic image understanding (PIU). The PIU generated consists of the most probable captions, parsed and represented by tree structures using NLP techniques, and detected objects. The PIU's are stored and indexed in a database system. For a query image, the Query Processor generates the most probable caption and parses it into the corresponding tree structure. Then an optimized tree-pattern query is constructed and executed on the database to retrieve a set of candidate images. The candidate images fetched are ranked using the tree-edit distance metric computed on the tree structures. Given a video, the Video Processor extracts a sequence of key scenes that are posed to the Query Processor to retrieve a set of candidate scenes. The candidate scene parse trees corresponding to a video are extracted and are ranked based on the number of matching scenes. We evaluated the performance of our system for large-scale image and video retrieval tasks on datasets containing everyday scenes and observed that our system could outperform state-ofthe- art techniques in terms of mean average precision.

A review of cross-modal retrieval for image-text

“…Secondly, with the exponential growth of cross-modal data, efficiently retrieving and querying relevant information from massive datasets across different modalities becomes an arduous task. In response to these challenges, researchers both domestically and internationally have conducted crossmodal retrieval research from various perspectives, yielding significant advancements in areas such as image-text retrieval and video-image-text retrieval [7,8,9,10,11]. This paper provides a comprehensive review of the latest research achievements over the past five years by taking image-text cross-modal retrieval as an example.…”