Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval

Abstract

How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

Learning With Batch-Wise Optimal Transport Loss for 3D Shape Recognition

“…Meanwhile, the retrieval performance is remarkably higher than the compared LBWR. We also compared our method with several mainstream approaches for 3D shape retrieval, including CDMR [22], SBR-VC [33], SP [51], FDC [51], Siamese network [58], DCML [19], DB-VLAT [55], and LWBR [63]. The evaluation criteria include NN, FT, ST, E, DCG, and mAP.…”

Deep Cross-Modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-Based 3D Shape Retrieval

“…In [30], the pyramid crossdomain neural networks were utilized to compensate for cross-domain divergences. In [1] and [25], Siamese metric networks are employed to minimize both within-modality and cross-modality intra-class distances whilst maximizing inter-class distances. In [25], the Wasserstein barycenters were additionally employed to aggregate multi-view deep features of rendered images from 3D models.…”

“…To characterize a 3D shape, we utilize the widely used multiview representation as in [18], [1], [24], i.e., projecting a 3D shape to N v grayscale images from N v rendered views that are evenly divided around the 3D shape. Thereafter, we can represent O as a batch of images I 2 =…”

“…z m i,j , and should be given higher importance during training. Existing deep metric learning based models [1], [25] equally treat all training samples.…”

“…However, due to the heterogeneity of sketches and 3D shapes, it is quite difficult to achieve modality-invariant discriminative representations. On the other hand, matching based methods focus on developing effective models for calculating similarities or distances between sketches and 3D shapes, among which deep metric learning based models [22], [1], [24] have achieved the state-of-the-art performance. Nevertheless, these methods fail to explore the varying importance of different training samples.…”