Towards Human Performance on Sketch-Based Image Retrieval

Abstract

Sketch-Based Image Retrieval (SBIR) is a crucial task in multimedia retrieval, where the goal is to retrieve a set of images that match a given sketch query. Researchers have already proposed several well-performing solutions for this task, but most focus on enhancing embedding through different approaches such as triplet loss, quadruplet loss, adding data augmentation, and using edge extraction. In this work, we tackle the problem from various angles. We start by examining the training data quality and show some of its limitations. Then, we introduce a Relative Triplet Loss (RTL), an adapted triplet loss to overcome those limitations through loss weighting based on anchors similarity. Through a series of experiments, we demonstrate that replacing a triplet loss with RTL outperforms previous state-of-the-art without the need for any data augmentation. In addition, we demonstrate why batch normalization is more suited for SBIR embeddings than l 2 -normalization and show that it improves significantly the performance of our models. We further investigate the capacity of models required for the photo and sketch domains and demonstrate that the photo encoder requires higher capacity than the sketch encoder, which validates the hypothesis formulated in [34]. Then, we propose a straightforward approach to train small models, such as ShuffleNetv2 [22] efficiently with a marginal loss of accuracy through knowledge distillation. The same approach used with larger models enabled us to outperform previous state-of-the-art results and achieve a recall of 62.38% at k = 1 on The Sketchy Database [30].

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

“…Zhang, et al [47] proposed a hybrid CNN structure consisting of A-Net and S-Net, with different branches dealing with appearance information and shape information respectively. Given that triplet networks are so popular in SBIR tasks, Seddati, et al [52] enhanced the SBIR pipeline performance by exploring multiple aspects, including embedding normalization, model sharing, margin selection, batch size, hard mining selection, and the number of hard triplets utilized during training. Furthermore, they introduced an innovative methodology for constructing SBIR solutions adaptable for deployment on low-power systems.…”

“…Although scientific research methods and reasonable retrieval strategies are used whenever possible, it is still possible to miss some valuable research. [18], [40], [42], [44], [47], [48], [49], [59], [62], [63], [64], [67], [72], [93], [102], [ [62], [63], [64] Sketchy [26] 75471sketches, 12500 photos 125 Public [26], [29], [38], [43], [44], [45], [47], [48], [49], [50], [51], [52], [55], [67], [71], [73], [74], [80], [86], [87], [92], [94], [99], [101], [104], [107], [108], [109], [110],…”

“…[62], [63] NDCG(Normalized Discounted Cumulative Gain) [66] precision@k [29], [32], [38], [55], [58], [61], [66], [84], [89], [92], [96], [97], [98], [99], [101], [103], [104], [105], [106], [107], [108], [109], [110], [111], [112], [113], [114], [115] precision-recall curve [18], [38], [40], [44], [56], [67], [72] precision/recall scores [93], [95] recall @ K [26], [43], [49], [52], [58], [86], [89], [92] retrieval accuracy [9], [10],…”