Image-text Retrieval: A Survey on Recent Research and Development

Abstract

Research in cognitive science has provided extensive evidence of human cognitive ability in performing physical reasoning of objects from noisy perceptual inputs. Such a cognitive ability is commonly known as intuitive physics. With advancements in deep learning, there is an increasing interest in building intelligent systems that are capable of performing physical reasoning from a given scene for the purpose of building better AI systems. As a result, many contemporary approaches in modelling intuitive physics for machine cognition have been inspired by literature from cognitive science. Despite the wide range of work in physical reasoning for machine cognition, there is a scarcity of reviews that organize and group these deep learning approaches. Especially at the intersection of intuitive physics and artificial intelligence, there is a need to make sense of the diverse range of ideas and approaches. Therefore, this paper presents a comprehensive survey of recent advances and techniques in intuitive physics-inspired deep learning approaches for physical reasoning. The survey will first categorize existing deep learning approaches into three facets of physical reasoning before organizing them into three general technical approaches and propose six categorical tasks of the field. Finally, we highlight the challenges of the current field and present some future research directions.

Understanding image-text relations and news values for multimodal news analysis

“…These models often have deep network backbones like BERT (Devlin et al, 2019 ) and are trained with loss functions such as masked language modeling, masked image region prediction and image-text matching. Recent work on image-text matching and retrieval (Cao et al, 2022 ) has explicitly focused on refining cross-attention mechanism and local alignment to get better retrieval performance. Consequently, these models focus more toward aligning literal (objective relations) relations than abstract and high-level semantic relations (semantic correlation STATUS, modification).…”

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

“…It also delves into experimental benchmarks, metrics, and performances, proffering novel ideas and recommendations for forthcoming research directions. Despite previous literature [7]- [10] addressing cross-modal retrieval, it is riddled with several deficiencies in timeliness, taxonomy, comprehensiveness, and more. Concretely, literature [7], [8] furnish insights into the early stages of cross-modal retrieval, yet their portrayal of representative methods and contemporary advancements is hindered by temporal gaps.…”

“…Notably, the recent emergence of Transformer architectures and vision-language pre-training models has wielded a profound impact on the domain of deep learning, fundamentally reshaping the cross-modal retrieval research landscape. While recent years have seen the publication of literature [9], [10], their scope and taxonomy fall markedly short. Within literature [9], the discourse on cross-modal retrieval methods rooted in self-attention mechanisms or large-scale pre-trained models is notably sparse.…”

“…Within literature [9], the discourse on cross-modal retrieval methods rooted in self-attention mechanisms or large-scale pre-trained models is notably sparse. Literature [10], on the other hand, centers predominantly on the field of image-text matching and fails to comprehensively synthesize methodologies for supervised cross-modal real-value retrieval and whole crossmodal hashing retrieval. Simultaneously, literature [9], [10] lumps deep learning-based cross-modal retrieval methodologies into an overly simplistic category, an approach ill-suited for the present advanced state of deep learning.…”

“…Literature [10], on the other hand, centers predominantly on the field of image-text matching and fails to comprehensively synthesize methodologies for supervised cross-modal real-value retrieval and whole crossmodal hashing retrieval. Simultaneously, literature [9], [10] lumps deep learning-based cross-modal retrieval methodologies into an overly simplistic category, an approach ill-suited for the present advanced state of deep learning. Within this realm, distinct deep techniques bear their core concepts, and consolidating them into a singular category obstructs a comprehensive grasp of the unique attributes inherent in divergent network architectures.…”