Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning

Abstract

In this work, we tackle the problem of single imagebased 3D shape retrieval (IBSR), where we seek to find the most matched shape of a given single 2D image from a shape repository. Most of the existing works learn to embed 2D images and 3D shapes into a common feature space and perform metric learning using a triplet loss. Inspired by the great success in recent contrastive learning works on self-supervised representation learning, we propose a novel IBSR pipeline leveraging contrastive learning. We note that adopting such cross-modal contrastive learning between 2D images and 3D shapes into IBSR tasks is non-trivial and challenging: contrastive learning requires very strong data augmentation in constructed positive pairs to learn the feature invariance, whereas traditional metric learning works do not have this requirement. Moreover, object shape and appearance are entangled in 2D query images, thus making the learning task more difficult than contrasting single-modal data. To mitigate the challenges, we propose to use multi-view grayscale rendered images from the 3D shapes as a shape representation. We then introduce a strong data augmentation technique based on color transfer, which can significantly but naturally change the appearance of the query image, effectively satisfying the need for contrastive learning. Finally, we propose to incorporate a novel category-level contrastive loss that helps distinguish similar objects from different categories, in addition to classic instance-level contrastive loss. Our experiments demonstrate that our approach achieves the best performance on * Corresponding Author is Lin Gao all the three popular IBSR benchmarks, including Pix3D, Stanford Cars, and Comp Cars, outperforming the previous state-of-the-art from 4% -15% on retrieval accuracy.

A systematic investigation on the surface properties of Ti2AlC via first-principles calculations

“…Datasets. To validate our proposed method, we perform experiments on three datasets: ModelNet40 [36], MI3DOR [43], Pix3D [24], and Pix3D with four categories (Pix3D-4, a subset of the Pix3D dataset created by [13]). The ModelNet40 dataset is a 3D object benchmark and contains…”

“…The advantage of using a set of 2D view representations is that they can directly employ the existing powerful CNNs for feature extraction [16,23] and reduce the domain gap between 3D models and images. Lin et al [13] used contrastive learning to realize instance-level 3D shape retrieval based on a single image. So far, great progress has been made in IBSR tasks.…”

“…The common embedding space aims to minimize the intra-class distance within the same category and maximize the inter-class distance between different categories. Recently, researchers have employed contrastive loss [9,13], cross-modal center loss [11], softmax cross-entropy loss [12], and other contrastive learning and metric learning methods to learn the common embedding space that characterizes multi-modal data features. To minimize the intra-class distance, researchers use the cross-modal center and cross-entropy loss to learn a common embedding space for all modality data by calculating the class center (mean of the feature) for different classes [11].…”

Cross-Modal Contrastive Learning for Domain Adaptation in 3D Semantic Segmentation

“…Contrastive learning has shown its powerful ability in selfsupervised training (Chen et al 2020;He et al 2020;Oord, Li, and Vinyals 2018), which facilitates representation learning by pulling features that are semantically similar and pushing away features that are semantically different. Recently, contrastive learning has been explored under several multi-modal learning scenarios, including vision-language (Radford et al 2021;Wen et al 2021;Yuan et al 2021;Zhang et al 2021;Bakkali et al 2022), video-text (Yang, Bisk, andZolfaghari et al 2021), image-point cloud (Lin et al 2021;Afham et al 2022;Liu et al 2021b) etc. By aligning multi-modal features, it takes advantage of modality complementary to improve model performance.…”

3D Model Retrieval Based on a 3D Shape Knowledge Graph

“…Consequently, different types of approaches have been proposed to handle this problem [1], [2], [3], [4], and several works have attempted to recover 3D information from 2D images (rendered view [5], [6], [7], [8], [9], scene [10], [11], [12], sketch [13], [14], [15]). In addition, some cross-modal 3D retrieval methods [16], [17], [18] are used to search and match the 3D models in databases, which reduces the difficulty of acquiring models, but still falls short of human expectations in terms of the accuracy and matching requirements.…”