Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences

Han, Zhizhong; Shang, Mingyang; Wang, Xiyang; Liu, Yu‐Shen; Zwicker, Matthias

doi:10.1609/aaai.v33i01.3301126

Cited by 52 publications

(41 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are also methods jointly learning features from point clouds and multi-view projections [47]. It is also possible to treat point clouds and views as sequences [26,17,15], or to use unsupervised learning [16].…”

Section: Related Workmentioning

confidence: 99%

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Pham

Hua

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

550

319

View full text Add to dashboard Cite

Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy (∼92%). Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background. Our dataset and code are publicly available in our project page 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Pham

Hua

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

550

319

View full text Add to dashboard Cite

show abstract

“…Deep learning models have led to significant progress in feature learning for 3D shapes [13,12,15,14,18,19,10,20,16,11]. Here, we focus on reviewing studies on point clouds.…”

Section: Related Workmentioning

confidence: 99%

Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds From Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction

Han

Wang

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

137

View full text Add to dashboard Cite

Unsupervised feature learning for point clouds has been vital for large-scale point cloud understanding. Recent deep learning based methods depend on learning global geometry from self-reconstruction. However, these methods are still suffering from ineffective learning of local geometry, which significantly limits the discriminability of learned features. To resolve this issue, we propose MAP-VAE to enable the learning of global and local geometry by jointly leveraging global and local self-supervision. To enable effective local self-supervision, we introduce multi-angle analysis for point clouds. In a multi-angle scenario, we first split a point cloud into a front half and a back half from each angle, and then, train MAP-VAE to learn to predict a back half sequence from the corresponding front half sequence. MAP-VAE performs this half-to-half prediction using RNN to simultaneously learn each local geometry and the spatial relationship among them. In addition, MAP-VAE also learns global geometry via self-reconstruction, where we employ a variational constraint to facilitate novel shape generation. The outperforming results in four shape analysis tasks show that MAP-VAE can learn more discriminative global or local features than the state-of-the-art methods.

show abstract

“…3D shape retrieval aims to match the relevant 3D shapes, which can be described as 3D mesh [9], voxel grid [8], point cloud [13] or multi-view [1,14], when given a query. The query can be 3D shape or other data representation modalities, such as: text [6], 2D image [3,[15][16][17] and 2D sketch [18], and we unite them as cross-domain 3D shape retrieval. When the query is 3D shape, many deep networks have been proposed to discover the intrinsic characteristics of 3D data, including the multi-view context and geometric structure information.…”

Section: Related Work 21 3d Shape Retrievalmentioning

confidence: 99%

“…With widespread use of 3D equipments and software [1][2][3], 3D shape retrieval, the task of matching the object (such as 2D image [4], 3D shape [5] or text [6]) from the gallery 3D shape dataset, has drawn especial attention from multimedia and computer vision communities. Despite the deep learning technique has already achieved good performance in 3D shape retrieval, there still exist some issues to be solved , especially for 2D image-based 3D shape retrieval (2D-to-3D) task.…”

Section: Introductionmentioning

confidence: 99%

Semantic Consistency Guided Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

Zhou

Nie

Song

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

2D image-based 3D shape retrieval (2D-to-3D) investigates the problem of matching the relevant 3D shapes from gallery dataset when given a query image. Recently, adversarial training and environmental style transfer learning have been successful applied to this task and achieved state-of-the-art performance. However, there still exist two problems. First, previous works only concentrate on the connection between the label and representation, where the unique visual characteristics of each instance are paid less attention. Second, the confused features or the transformed images can only cheat the discriminator but can not guarantee the semantic consistency. In another words, features of 2D desk may be mapped nearby the features of 3D chair. In this paper, we propose a novel semantic consistency guided instance feature alignment network (SC-IFA) to address these limitations. SC-IFA mainly consists of two parts, instance visual feature extraction and cross-domain instance feature adaptation. For the first module, unlike previous methods, which merely employ 2D CNN to extract the feature, we additionally maximize the mutual information between the input and feature to enhance the capability of feature representation for each instance. For the second module, we first introduce the margin disparity discrepancy model to mix up the cross-domain features in an adversarial training way. Then, we design two feature translators to transform the feature from one domain to another domain, and impose the translation loss and correlation loss on the transformed features to preserve the semantic consistency. Extensive experimental results on two benchmarks, MI3DOR and MI3DOR-2, verify SC-IFA is superior to the state-of-the-art methods.

show abstract

Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences

Cited by 52 publications

References 6 publications

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds From Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction

Semantic Consistency Guided Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

Contact Info

Product

Resources

About