Deep cross-modal discriminant adversarial learning for zero-shot sketch-based image retrieval

Jiao, Shichao; Xiong, Fengguang; Yang, Xiaowen; Han, Huiyan; He, Ligang; Kuang, Liqun

doi:10.1007/s00521-022-07169-6

Cited by 6 publications

(5 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To verify the effectiveness of the proposed method in this study, we compared the proposed model with the following baseline deep learning methods for cross-modal retrieval: MR-SBIR [8], DAL [19], DSM [70], DOODLE [71], DSCMR [72], CMCL [73], and ACNet [18].…”

Section: Experiments On Rsketch Datasetmentioning

confidence: 99%

“…To bridge the two modalities, a common approach contains three stages: feature extraction from both modalities, feature enhancement, and image retrieval. The latest solutions, like ACNet [18], DAL [19], and ZSE-SBIR [13], often employ deep structures like ViT (vision transformer) and Resnet and use homogeneous [20], Siamese branch [9], or heterogeneous structures [21] for different modalities. Despite the inspiring and promising development of SBIR, only a small number of recent research publications have been dedicated to remote-sensing SBIR [8,[22][23][24][25][26].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Yang,

Wang,

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

Effectively and efficiently retrieving images from remote-sensing databases is a critical challenge in the realm of remote-sensing big data. Utilizing hand-drawn sketches as retrieval inputs offers intuitive and user-friendly advantages, yet the potential of multi-level feature integration from sketches remains underexplored, leading to suboptimal retrieval performance. To address this gap, our study introduces a novel zero-shot, sketch-based retrieval method for remote-sensing images, leveraging multi-level feature extraction, self-attention-guided tokenization and filtering, and cross-modality attention update. This approach employs only vision information and does not require semantic knowledge concerning the sketch and image. It starts by employing multi-level self-attention guided feature extraction to tokenize the query sketches, as well as self-attention feature extraction to tokenize the candidate images. It then employs cross-attention mechanisms to establish token correspondence between these two modalities, facilitating the computation of sketch-to-image similarity. Our method significantly outperforms existing sketch-based remote-sensing image retrieval techniques, as evidenced by tests on multiple datasets. Notably, it also exhibits robust zero-shot learning capabilities in handling unseen categories and strong domain adaptation capabilities in handling unseen novel remote-sensing data. The method’s scalability can be further enhanced by the pre-calculation of retrieval tokens for all candidate images in a database. This research underscores the significant potential of multi-level, attention-guided tokenization in cross-modal remote-sensing image retrieval. For broader accessibility and research facilitation, we have made the code and dataset used in this study publicly available online.

show abstract

Section: Experiments On Rsketch Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Yang,

Wang,

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Deep learning SBIR algorithms are computationally demanding. Utilizing Graphics Processing Units (GPUs) can significantly accelerate computation compared to traditional Central Processing Units (CPUs) due to their parallel processing capabilities [96], [115]. SBIR systems typically require significant memory resources, especially with large image databases.…”

Section: Rq3: What Are the Various Metrics For Evaluating The Perform...mentioning

confidence: 99%

“…[82] mAP [17], [18], [32], [38], [40], [41], [42], [44], [45], [46], [47], [48], [49], [51], [56], [59], [60], [61], [64], [66], [67], [69], [72], [84], [91], [93], [96], [97], [98], [99], [100], [101], [102], [103], [104], [105], [107], [108], [109], [110], [111], [112], [113], [114], [115] memory loads [38], [58], [61]…”

Section: Ap (Average Precision)mentioning

confidence: 99%

“…[62], [63] NDCG(Normalized Discounted Cumulative Gain) [66] precision@k [29], [32], [38], [55], [58], [61], [66], [84], [89], [92], [96], [97], [98], [99], [101], [103], [104], [105], [106], [107], [108], [109], [110], [111], [112], [113], [114], [115] precision-recall curve [18], [38], [40], [44], [56], [67], [72] precision/recall scores [93], [95] recall @ K [26], [43], [49], [52], [58], [86], [89], [92] retrieval accuracy [9], [10],…”

Section: Mrp (Mean Rank Precision)mentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

Yang,

Ismail,

Pang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Sketch-based image retrieval (SBIR) utilizes sketches to search for images containing similar objects or scenes. Due to the proliferation of touch-screen devices, sketching has become more accessible and therefore has received increasing attention. Deep learning has emerged as a potential tool for SBIR, allowing models to automatically extract image features and learn from large amounts of data. To the best of our knowledge, there is currently no systematic literature review (SLR) of SBIR with deep learning. Therefore, the aim of this review is to incorporate related works into a systematic study, highlighting the main contributions of individual researchers over the years, with a focus on past, present and future trends. To achieve the purpose of this study, 90 studies from 2016 to June 2023 in 4 databases were collected and analyzed using the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) framework. The specific models, datasets, evaluation metrics, and applications of deep learning in SBIR are discussed in detail. This study found that Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) are the most widely used deep learning methods for SBIR. A commonly used dataset is Sketchy, especially in the latest Zero-shot sketch-based image retrieval (ZS-SBIR) task. The results show that Mean Average Precision (mAP) is the most commonly used metric for quantitative evaluation of SBIR. Finally, we provide some future directions and guidance for researchers based on the results of this review.INDEX TERMS Sketch-based image retrieval, SBIR, SLR, PRISMA, deep learning.

show abstract

Discriminative latent semantics-preserving similarity embedding hashing for cross-modal retrieval

Chen,

Tan,

Yang

et al. 2024

Neural Comput & Applic

View full text Add to dashboard Cite

Deep cross-modal discriminant adversarial learning for zero-shot sketch-based image retrieval

Cited by 6 publications

References 52 publications

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

Discriminative latent semantics-preserving similarity embedding hashing for cross-modal retrieval

Contact Info

Product

Resources

About