Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval

Ling, Zhixin; Xing, Zhen; Li, Jiangtong; Niu, Li

doi:10.1145/3503161.3548147

Cited by 10 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In their proposition of the Multi-Level Region Matching model (MLRM) [87] , two integral components contribute to its architecture. The initial segment involves the Discriminative Region Extraction module (DRE), responsible for extracting multi-level CNN-based features.…”

Section: ) Approaches Used In Fine-grained Sbirmentioning

confidence: 99%

“…Although scientific research methods and reasonable retrieval strategies are used whenever possible, it is still possible to miss some valuable research. [18], [40], [42], [44], [47], [48], [49], [59], [62], [63], [64], [67], [72], [93], [102], [ [62], [63], [64] Sketchy [26] 75471sketches, 12500 photos 125 Public [26], [29], [38], [43], [44], [45], [47], [48], [49], [50], [51], [52], [55], [67], [71], [73], [74], [80], [86], [87], [92], [94], [99], [101], [104], [107], [108], [109], [110],…”

Section: ) Approaches Used In Zero-shot Sbirmentioning

confidence: 99%

“…[87] QMUL-ShoeV2 [80] 2,000 image-sketch pairs 1 Public [30], [71], [74], [78], [82], [87], [90], [91], [94] QMUL Chair [10] 297 sketches, 297 photos 1 Public [10], [71], [79], [81], [83], [87] QMUL-ChairV2 [10] 400 image-sketch pairs 1 Public [30], [74], [78], [82], [87], [90], [91] QMUL Handbag [83] 568 sketches, 568 photos 1 Public [73], [80], [81], [83…”

Section: ) Approaches Used In Zero-shot Sbirmentioning

confidence: 99%

See 2 more Smart Citations

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

Yang,

Ismail,

Pang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Sketch-based image retrieval (SBIR) utilizes sketches to search for images containing similar objects or scenes. Due to the proliferation of touch-screen devices, sketching has become more accessible and therefore has received increasing attention. Deep learning has emerged as a potential tool for SBIR, allowing models to automatically extract image features and learn from large amounts of data. To the best of our knowledge, there is currently no systematic literature review (SLR) of SBIR with deep learning. Therefore, the aim of this review is to incorporate related works into a systematic study, highlighting the main contributions of individual researchers over the years, with a focus on past, present and future trends. To achieve the purpose of this study, 90 studies from 2016 to June 2023 in 4 databases were collected and analyzed using the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) framework. The specific models, datasets, evaluation metrics, and applications of deep learning in SBIR are discussed in detail. This study found that Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) are the most widely used deep learning methods for SBIR. A commonly used dataset is Sketchy, especially in the latest Zero-shot sketch-based image retrieval (ZS-SBIR) task. The results show that Mean Average Precision (mAP) is the most commonly used metric for quantitative evaluation of SBIR. Finally, we provide some future directions and guidance for researchers based on the results of this review.INDEX TERMS Sketch-based image retrieval, SBIR, SLR, PRISMA, deep learning.

show abstract

Section: ) Approaches Used In Fine-grained Sbirmentioning

confidence: 99%

Section: ) Approaches Used In Zero-shot Sbirmentioning

confidence: 99%

Section: ) Approaches Used In Zero-shot Sbirmentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

Yang,

Ismail,

Pang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Fine-grained image retrieval can be divided into the content-based method [34,42,43] and sketch-based method [19,23,24] according to the type of query image. Among them, the content-based method is committed to retrieving images of various sub-categories with only subtle differences.…”

Section: Related Work 21 Content-based Fine-grained Retrievalmentioning

confidence: 99%

Attributes Grouping and Mining Hashing for Fine-Grained Image Retrieval

Lu,

Chen,

Cao

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

In recent years, hashing methods have been popular in the largescale media search for low storage and strong representation capabilities. To describe objects with similar overall appearance but subtle differences, more and more studies focus on hashing-based fine-grained image retrieval. Existing hashing networks usually generate both local and global features through attention guidance on the same deep activation tensor, which limits the diversity of feature representations. To handle this limitation, we substitute convolutional descriptors for attention-guided features and propose an Attributes Grouping and Mining Hashing (AGMH), which groups and embeds the category-specific visual attributes in multiple descriptors to generate a comprehensive feature representation for efficient fine-grained image retrieval. Specifically, an Attention Dispersion Loss (ADL) is designed to force the descriptors to attend to various local regions and capture diverse subtle details. Moreover, we propose a Stepwise Interactive External Attention (SIEA) to mine critical attributes in each descriptor and construct correlations between fine-grained attributes and objects. The attention mechanism is dedicated to learning discrete attributes, which will not cost additional computations in hash codes generation. Finally, the compact binary codes are learned by preserving pairwise similarities. Experimental results demonstrate that AGMH consistently

show abstract

“…Recall@1(%) Chance [30] 0.01 Sketch me that shoe [45] 25.87 Siamese Network [30] 27.36 Triplet Network [30] 37.10 Quadruplet MT [32] 38.21 DCCRM(S+I) [41] 40.16 DeepTCNet [21] 40.81 Triplet attention [33] 41.66 Quadruplet MT v2 [32] 42.16 LA [43] 43.1 DCCRM(S+I+D) [41] 46.20 Human [30] 54.27 ResNet18 [8] 45.95 DCCRM [41] 46.20 ResNet50 [8] 52.19 ResNet18 [34] 52.75 ResNet18 [35] 53.61 ResNet101 [8] 54.59 DLA [43] 54.9 ResNet18 2×2 [35] 55.10 Our R18 RT L 55.27 ResNet50 [35] 56.29 Our R18 RT L+BN 57.20 MLRM [18] 57.20 ResNet34 [35] 57.43 ResNet34 2×2 [35] 58.23 ResNet50 2×2 [35] 58.37 Our R34 RT L 58.50 Our Shuf f leN etV 2 Huber+DG 59.1 Our R34 RT L+BN 59.99 VT [35] 62.25 Our R152 Huber 62.38…”

Section: Modelmentioning

confidence: 99%

Towards Human Performance on Sketch-Based Image Retrieval

Seddati¹,

Dupont²,

Mahmoudi

et al. 2022

International Conference on Content-Based Multimedia Indexing

View full text Add to dashboard Cite

Sketch-Based Image Retrieval (SBIR) is a crucial task in multimedia retrieval, where the goal is to retrieve a set of images that match a given sketch query. Researchers have already proposed several well-performing solutions for this task, but most focus on enhancing embedding through different approaches such as triplet loss, quadruplet loss, adding data augmentation, and using edge extraction. In this work, we tackle the problem from various angles. We start by examining the training data quality and show some of its limitations. Then, we introduce a Relative Triplet Loss (RTL), an adapted triplet loss to overcome those limitations through loss weighting based on anchors similarity. Through a series of experiments, we demonstrate that replacing a triplet loss with RTL outperforms previous state-of-the-art without the need for any data augmentation. In addition, we demonstrate why batch normalization is more suited for SBIR embeddings than l 2 -normalization and show that it improves significantly the performance of our models. We further investigate the capacity of models required for the photo and sketch domains and demonstrate that the photo encoder requires higher capacity than the sketch encoder, which validates the hypothesis formulated in [34]. Then, we propose a straightforward approach to train small models, such as ShuffleNetv2 [22] efficiently with a marginal loss of accuracy through knowledge distillation. The same approach used with larger models enabled us to outperform previous state-of-the-art results and achieve a recall of 62.38% at k = 1 on The Sketchy Database [30].

show abstract

Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval

Cited by 10 publications

References 25 publications

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

Attributes Grouping and Mining Hashing for Fine-Grained Image Retrieval

Towards Human Performance on Sketch-Based Image Retrieval

Contact Info

Product

Resources

About